Sep 09 14:09:02.162267 ip-10-0-139-251 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 14:09:02.162275 ip-10-0-139-251 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 14:09:02.162281 ip-10-0-139-251 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 14:09:02.162434 ip-10-0-139-251 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 14:09:12.411427 ip-10-0-139-251 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 14:09:12.411447 ip-10-0-139-251 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4bde589563c145699181ed779d1c8dff --
Sep 09 14:11:06.122354 ip-10-0-139-251 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 14:11:06.519866 ip-10-0-139-251 kubenswrapper[2532]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 14:11:06.519866 ip-10-0-139-251 kubenswrapper[2532]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 14:11:06.519866 ip-10-0-139-251 kubenswrapper[2532]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 14:11:06.519866 ip-10-0-139-251 kubenswrapper[2532]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 14:11:06.519866 ip-10-0-139-251 kubenswrapper[2532]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 14:11:06.520449 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.519937    2532 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 14:11:06.525719 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525704    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 14:11:06.525719 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525717    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525722    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525726    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525730    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525733    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525736    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525739    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525742    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525744    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525747    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525750    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525752    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525755    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525758    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525760    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525763    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525773    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525776    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525778    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525781    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 14:11:06.525781 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525783    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525786    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525789    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525792    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525794    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525798    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525800    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525803    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525805    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525808    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525810    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525813    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525815    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525818    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525820    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525823    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525825    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525828    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525831    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525833    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 14:11:06.526277 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525836    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525838    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525841    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525843    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525846    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525848    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525851    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525853    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525856    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525859    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525861    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525865    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525868    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525872    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525877    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525880    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525883    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525886    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525888    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 14:11:06.526795 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525891    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525894    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525897    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525899    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525902    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525904    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525907    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525909    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525912    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525915    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525917    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525926    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525928    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525931    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525934    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525936    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525939    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525941    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525944    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 14:11:06.527274 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525950    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525954    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525957    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525960    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525963    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525966    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525969    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525971    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525974    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525977    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525980    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525982    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525985    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525988    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525990    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525994    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.525998    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526001    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526004    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526006    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 14:11:06.527733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526697    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526703    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526707    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526710    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526712    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526716    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526719    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526722    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526725    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526727    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526730    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526734    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526739    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526742    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526745    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526747    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526751    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526754    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526757    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 14:11:06.528225 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526760    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526763    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526766    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526769    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526771    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526774    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526776    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526780    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526783    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526786    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526789    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526792    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526796    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526799    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526801    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526804    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526807    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526810    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526813    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 14:11:06.528695 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526815    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526818    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526820    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526825    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526827    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526830    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526833    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526835    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526838    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526840    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526843    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526845    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526848    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526850    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526852    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526855    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526858    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526860    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526863    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526866    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 14:11:06.529205 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526868    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526878    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526881    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526884    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526886    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526889    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526892    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526895    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526897    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526900    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526905    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526908    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526910    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526914    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526916    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526920    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526923    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526926    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526929    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526931    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 14:11:06.529692 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526934    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526936    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526939    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526942    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526944    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526947    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526949    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526952    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526955    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526957    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526960    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526962    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526965    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526968    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526970    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526973    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526976    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526978    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526981    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526983    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 14:11:06.530194 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.526986    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527333    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527338    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527340    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527343    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527346    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527349    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527352    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527355    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527358    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527361    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527363    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527368    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527372    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527375    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527378    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527381    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527383    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527386    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 14:11:06.530678 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527389    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527391    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527394    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527397    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527399    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527402    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527404    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527407    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527409    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527412    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527414    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527417    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527419    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527422    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527424    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527427    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527431    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527434    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527437    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 14:11:06.531173 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527441    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527443    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527446    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527449    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527452    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527454    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527457    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527460    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527463    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527466    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527468    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527471    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527473    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527476    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527478    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527480    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527483    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527486    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527488    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527491    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 14:11:06.531623 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527494    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527496    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527499    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527501    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527504    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527506    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527509    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527511    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527514    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527517    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527520    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527522    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527525    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527528    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527531    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527533    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527536    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527539    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527542    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527544    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527547    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 14:11:06.532119 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527549    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527552    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527556    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527559    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527561    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527564    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527566    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527568    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527571    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527573    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527576    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527578    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527580    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527583    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527586    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527590    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527592    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527595    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527598    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527600    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 14:11:06.532627 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.527603    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527678    2532 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527722    2532 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527728    2532 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527732    2532 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527738    2532 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527741    2532 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527745    2532 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527749    2532 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527752    2532 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527756    2532 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527759    2532 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527762    2532 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527765    2532 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527770    2532 flags.go:64] FLAG: --cgroup-root=""
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527773    2532 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527776    2532 flags.go:64] FLAG: --client-ca-file=""
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527779    2532 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527782    2532 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527786    2532 flags.go:64] FLAG: --cluster-domain=""
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527789    2532 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527792    2532 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527795    2532 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527799    2532 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 14:11:06.533120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527802    2532 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527805    2532 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527808    2532 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527811    2532 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527814    2532 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527833    2532 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527837    2532 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527841    2532 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527845    2532 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527849    2532 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527852    2532 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527855    2532 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527858    2532 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527861    2532 flags.go:64] FLAG: --enable-server="true"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527865    2532 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527869    2532 flags.go:64] FLAG: --event-burst="100"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527872    2532 flags.go:64] FLAG: --event-qps="50"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527875    2532 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527878    2532 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527881    2532 flags.go:64] FLAG: --eviction-hard=""
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527884    2532 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527887    2532 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527891    2532 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527894    2532 flags.go:64] FLAG: --eviction-soft=""
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527897    2532 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 14:11:06.533705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527900    2532 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527903    2532 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527905    2532 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527908    2532 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527911    2532 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527914    2532 flags.go:64] FLAG: --feature-gates=""
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527917    2532 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527920    2532 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527923    2532 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527926    2532 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527929    2532 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527932    2532 flags.go:64] FLAG: --help="false"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527935    2532 flags.go:64] FLAG: --hostname-override="ip-10-0-139-251.ec2.internal"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527938    2532 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527940    2532 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527946    2532 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527949    2532 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527953    2532 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527956    2532 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527958    2532 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527961    2532 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527964    2532 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527967    2532 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527970    2532 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 14:11:06.534366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527973    2532 flags.go:64] FLAG: --kube-reserved=""
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527976    2532 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527979    2532 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527982    2532 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527985    2532 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527989    2532 flags.go:64] FLAG: --lock-file=""
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527991    2532 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527994    2532 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.527997    2532 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528002    2532 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528005    2532 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528007    2532 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528010    2532 flags.go:64] FLAG: --logging-format="text"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528013    2532 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528016    2532 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528019    2532 flags.go:64] FLAG: --manifest-url=""
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528021    2532 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528027    2532 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528030    2532 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528034    2532 flags.go:64] FLAG: --max-pods="110"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528037    2532 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528040    2532 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528042    2532 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528047    2532 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528050    2532 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 14:11:06.534970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528053    2532 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528056    2532 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528063    2532 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528066    2532 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528069    2532 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528072    2532 flags.go:64] FLAG: --pod-cidr=""
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528075    2532 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528081    2532 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528083    2532 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528086    2532 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528090    2532 flags.go:64] FLAG: --port="10250"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528093    2532 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528108    2532 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0af8d4a1458e3230c"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528111    2532 flags.go:64] FLAG: --qos-reserved=""
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528114    2532 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528117    2532 flags.go:64] FLAG: --register-node="true"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528120    2532 flags.go:64] FLAG: --register-with-taints=""
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528124    2532 flags.go:64] FLAG: --registry-burst="10"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528127    2532 flags.go:64] FLAG: --registry-qps="5"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528129    2532 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528132    2532 flags.go:64] FLAG: --reserved-memory=""
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528136    2532 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528139    2532 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528142    2532 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528145    2532 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 14:11:06.535589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528148    2532 flags.go:64] FLAG: --runonce="false"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528151    2532 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528154    2532 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528157    2532 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528160    2532 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528162    2532 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528167    2532 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528170    2532 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528173    2532 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528176    2532 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528178    2532 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528181    2532 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528184    2532 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528187    2532 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528191    2532 flags.go:64] FLAG: --system-cgroups=""
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528193    2532 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528199    2532 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528201    2532 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528204    2532 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528210    2532 flags.go:64] FLAG: --tls-min-version=""
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528212    2532 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528215    2532 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528218    2532 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528221    2532 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528224    2532 flags.go:64] FLAG: --v="2"
Sep 09 14:11:06.536216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528228    2532 flags.go:64] FLAG: --version="false"
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528232    2532 flags.go:64] FLAG: --vmodule=""
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528236    2532 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.528239    2532 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528347    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528351    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528354    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528357    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528360    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528362    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528365    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528367    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528370    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528374    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528377    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528380    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528382    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528385    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528388    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528390    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 14:11:06.536841 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528393    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528397    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528401    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528403    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528406    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528409    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528412    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528415    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528418    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528420    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528423    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528425    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528429    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528433    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528435    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528438    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528441    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528444    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528447    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 14:11:06.537479 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528450    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528453    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528455    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528458    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528461    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528463    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528467    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528469    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528472    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528475    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528477    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528480    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528483    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528487    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528489    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528492    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528495    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528498    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528500    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 14:11:06.538283 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528503    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528506    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528509    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528512    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528514    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528524    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528527    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528529    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528532    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528534    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528537    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528539    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528542    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528544    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528547    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528550    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528552    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528555    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528557    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528561    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 14:11:06.538885 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528564    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528566    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528569    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528572    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528574    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528577    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528579    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528582    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528585    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528588    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528591    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528593    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528597    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528599    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528602    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528605    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528607    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528610    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528612    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528615    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 14:11:06.539412 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528617    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528620    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528623    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528625    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.528628    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.529433    2532 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.536185    2532 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.536199    2532 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.536211    2532 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.536217    2532 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536268    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536272    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536276    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536280    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 14:11:06.539895 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536283    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536285    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536289    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536292    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536295    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536297    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536300    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536303    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536306    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536308    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536311    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536313    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536316    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536319    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536321    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536324    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536326    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536329    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536331    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536334    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 14:11:06.540270 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536337    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536340    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536342    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536345    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536347    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536351    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536353    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536356    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536359    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536361    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536364    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536368    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536372    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536375    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536377    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536380    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536382    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536385    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536388    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 14:11:06.540760 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536391    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536393    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536396    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536398    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536401    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536404    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536406    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536409    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536412    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536414    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536417    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536420    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536422    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536425    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536428    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536431    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536433    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536436    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536439    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536442    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 14:11:06.541314 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536445    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536448    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536450    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536453    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536455    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536458    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536460    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536463    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536465    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536468    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536470    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536473    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536475    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536478    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536480    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536483    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536486    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536488    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536492    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536495    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 14:11:06.541802 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536498    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536500    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536503    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536507    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536510    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536513    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536516    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536520    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536523    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536526    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536529    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536532    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536535    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536538    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536541    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 14:11:06.542392 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536544    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.536549    2532 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536651    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536656    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536658    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536661    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536664    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536667    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536670    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536672    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536675    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536677    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536680    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536682    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536685    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 14:11:06.542756 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536688    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536690    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536693    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536695    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536698    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536701    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536703    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536706    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536709    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536711    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536714    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536716    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536719    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536722    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536724    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536727    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536730    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536732    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536735    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 14:11:06.543168 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536737    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536740    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536742    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536745    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536747    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536749    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536752    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536756    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536760    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536763    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536766    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536769    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536772    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536775    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536777    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536780    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536783    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536786    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536789    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 14:11:06.543629 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536791    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536794    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536796    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536799    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536802    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536804    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536807    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536809    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536812    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536814    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536817    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536820    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536822    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536825    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536827    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536830    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536832    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536835    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536837    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536840    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 14:11:06.544084 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536842    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536845    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536847    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536850    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536853    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536856    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536859    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536861    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536864    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536866    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536869    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536872    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536874    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536876    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536879    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536881    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536884    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536886    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536889    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536891    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 14:11:06.544648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536894    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536896    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536898    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536901    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536904    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536906    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536909    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:06.536911    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.536916    2532 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.537008    2532 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.540041    2532 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.540879    2532 server.go:1019] "Starting client certificate rotation"
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.540974    2532 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 14:11:06.545200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.541008    2532 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 14:11:06.545590 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.543155    2532 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 14:11:06.545590 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.544221    2532 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 14:11:06.557956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.557936    2532 log.go:25] "Validated CRI v1 runtime API"
Sep 09 14:11:06.560568 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.560553    2532 log.go:25] "Validated CRI v1 image API"
Sep 09 14:11:06.561306 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.561291    2532 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 14:11:06.564492 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.564475    2532 fs.go:135] Filesystem UUIDs: map[2fecc994-4ff5-4196-801e-5fbbdb1184b8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ad2f3369-67f0-42b9-8388-ba8da7a962b2:/dev/nvme0n1p3]
Sep 09 14:11:06.564552 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.564491    2532 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 14:11:06.568909 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.568820    2532 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 14:11:06.569742 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.569617    2532 manager.go:216] Machine: {Timestamp:2026-09-09 14:11:06.568359145 +0000 UTC m=+0.331154378 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108234 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af3a9590313901e9ed23776cf67b5 SystemUUID:ec2af3a9-5903-1390-1e9e-d23776cf67b5 BootID:4bde5895-63c1-4569-9181-ed779d1c8dff Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7b:1d:1f:df:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7b:1d:1f:df:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:92:9a:f1:00:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 14:11:06.569742 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.569742    2532 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 14:11:06.569831 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.569810    2532 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 14:11:06.573499 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.573461    2532 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 14:11:06.573628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.573503    2532 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 14:11:06.574570 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.574559    2532 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 14:11:06.574610 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.574572    2532 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 14:11:06.574610 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.574580    2532 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 14:11:06.575236 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.575226    2532 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 14:11:06.576618 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.576608    2532 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 14:11:06.579812 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.579801    2532 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 14:11:06.579924 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.579915    2532 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 14:11:06.579953 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.579944    2532 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 14:11:06.579980 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.579955    2532 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 14:11:06.579980 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.579967    2532 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 14:11:06.579980 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.579976    2532 kubelet.go:398] "Adding apiserver pod source"
Sep 09 14:11:06.580083 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.579984    2532 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 14:11:06.580254 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.580241    2532 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 14:11:06.580316 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.580260    2532 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 14:11:06.582292 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.582272    2532 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 14:11:06.584750 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.584735    2532 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 14:11:06.584812 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.584766    2532 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 14:11:06.585583 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585571    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 14:11:06.585625 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585588    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 14:11:06.585625 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585594    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 14:11:06.585625 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585603    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 14:11:06.585625 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585611    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 14:11:06.585625 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585616    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 14:11:06.585754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585628    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 14:11:06.585754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585633    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 14:11:06.585754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585639    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 14:11:06.585754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585645    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 14:11:06.585754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585652    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 14:11:06.585754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.585658    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 14:11:06.586448 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.586437    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 14:11:06.586448 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.586449    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 14:11:06.589853 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.589830    2532 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 14:11:06.590012 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.589998    2532 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 14:11:06.590071 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.590019    2532 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 14:11:06.590211 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.590199    2532 server.go:1268] "Started kubelet"
Sep 09 14:11:06.590277 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.590252    2532 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 14:11:06.590326 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.590262    2532 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 14:11:06.590326 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.590322    2532 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 14:11:06.590850 ip-10-0-139-251 systemd[1]: Started Kubernetes Kubelet.
Sep 09 14:11:06.591432 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.591412    2532 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 14:11:06.591999 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.591985    2532 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 14:11:06.595556 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.595541    2532 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 14:11:06.596460 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.596427    2532 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t9tzb"
Sep 09 14:11:06.597751 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.597734    2532 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 14:11:06.598411 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.598394    2532 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 14:11:06.598508 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.598495    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:06.598961 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.598083    2532 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-251.ec2.internal.18d3ac6e4466a117  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-251.ec2.internal,UID:ip-10-0-139-251.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-251.ec2.internal,},FirstTimestamp:2026-09-09 14:11:06.590175511 +0000 UTC m=+0.352970742,LastTimestamp:2026-09-09 14:11:06.590175511 +0000 UTC m=+0.352970742,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-251.ec2.internal,}"
Sep 09 14:11:06.599080 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599061    2532 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 14:11:06.599193 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599138    2532 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 14:11:06.599193 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599154    2532 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 14:11:06.599291 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599221    2532 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 14:11:06.599291 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599230    2532 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 14:11:06.599291 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599281    2532 factory.go:55] Registering systemd factory
Sep 09 14:11:06.599415 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599295    2532 factory.go:223] Registration of the systemd container factory successfully
Sep 09 14:11:06.599528 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599512    2532 factory.go:153] Registering CRI-O factory
Sep 09 14:11:06.599582 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599534    2532 factory.go:223] Registration of the crio container factory successfully
Sep 09 14:11:06.599614 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599585    2532 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 14:11:06.599614 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599602    2532 factory.go:103] Registering Raw factory
Sep 09 14:11:06.599666 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599616    2532 manager.go:1196] Started watching for new ooms in manager
Sep 09 14:11:06.600013 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.599999    2532 manager.go:349] Starting recovery of all containers
Sep 09 14:11:06.605440 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.605401    2532 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t9tzb"
Sep 09 14:11:06.609500 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.608717    2532 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 14:11:06.609500 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.608740    2532 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 14:11:06.609717 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.609698    2532 manager.go:354] Recovery completed
Sep 09 14:11:06.613875 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.613861    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 14:11:06.615677 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.615664    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 14:11:06.615765 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.615695    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 14:11:06.615765 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.615710    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientPID"
Sep 09 14:11:06.617173 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.616495    2532 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 14:11:06.617173 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.616507    2532 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 14:11:06.617173 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.616520    2532 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 14:11:06.617173 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.616549    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 14:11:06.618352 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.617680    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 14:11:06.618352 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.617706    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 14:11:06.618352 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.617720    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientPID"
Sep 09 14:11:06.618503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.618411    2532 policy_none.go:49] "None policy: Start"
Sep 09 14:11:06.618503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.618424    2532 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 14:11:06.618503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.618436    2532 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 14:11:06.620201 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.620190    2532 policy_none.go:47] "Start"
Sep 09 14:11:06.643697 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.643681    2532 manager.go:337] "Starting Device Plugin manager"
Sep 09 14:11:06.643780 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.643716    2532 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 14:11:06.643780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.643730    2532 server.go:87] "Starting device plugin registration server"
Sep 09 14:11:06.643979 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.643969    2532 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 14:11:06.644011 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.643981    2532 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 14:11:06.644124 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.644089    2532 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 14:11:06.644224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.644176    2532 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 14:11:06.644224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.644185    2532 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 14:11:06.644615 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.644595    2532 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 14:11:06.644681 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.644642    2532 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:06.726458 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.726433    2532 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 14:11:06.727588 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.727569    2532 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 14:11:06.727654 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.727610    2532 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 14:11:06.727654 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.727629    2532 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 14:11:06.727749 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.727666    2532 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 14:11:06.729716 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.729696    2532 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 14:11:06.744518 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.744500    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 14:11:06.745550 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.745532    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 14:11:06.745633 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.745560    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 14:11:06.745633 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.745570    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientPID"
Sep 09 14:11:06.745633 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.745588    2532 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-251.ec2.internal"
Sep 09 14:11:06.751718 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.751701    2532 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-251.ec2.internal"
Sep 09 14:11:06.751790 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.751722    2532 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-251.ec2.internal\": node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:06.773735 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.773684    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:06.828234 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.828215    2532 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal"]
Sep 09 14:11:06.828301 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.828276    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 14:11:06.829569 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.829554    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 14:11:06.829642 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.829583    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 14:11:06.829642 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.829593    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientPID"
Sep 09 14:11:06.830869 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.830856    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 14:11:06.830993 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.830975    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:06.831039 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.831020    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 14:11:06.831985 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.831961    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 14:11:06.831985 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.831986    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 14:11:06.832151 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.832002    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientPID"
Sep 09 14:11:06.832151 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.832007    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 14:11:06.832151 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.832024    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 14:11:06.832151 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.832034    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientPID"
Sep 09 14:11:06.833216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.833201    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:06.833292 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.833228    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 14:11:06.833905 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.833887    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 14:11:06.833989 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.833918    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 14:11:06.833989 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:06.833932    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeHasSufficientPID"
Sep 09 14:11:06.862618 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.862601    2532 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-251.ec2.internal\" not found" node="ip-10-0-139-251.ec2.internal"
Sep 09 14:11:06.866703 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.866688    2532 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-251.ec2.internal\" not found" node="ip-10-0-139-251.ec2.internal"
Sep 09 14:11:06.874741 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.874725    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:06.975232 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:06.975206    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:07.000516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.000498    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d1e80e0bdfbb3cd0f5d4bdeeaa8f968-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal\" (UID: \"4d1e80e0bdfbb3cd0f5d4bdeeaa8f968\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.000595 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.000522    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d1e80e0bdfbb3cd0f5d4bdeeaa8f968-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal\" (UID: \"4d1e80e0bdfbb3cd0f5d4bdeeaa8f968\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.000595 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.000538    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6aac1761331084f3b830a859c99fcd9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-251.ec2.internal\" (UID: \"d6aac1761331084f3b830a859c99fcd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.075765 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.075726    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:07.101237 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.101217    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d1e80e0bdfbb3cd0f5d4bdeeaa8f968-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal\" (UID: \"4d1e80e0bdfbb3cd0f5d4bdeeaa8f968\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.101280 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.101242    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d1e80e0bdfbb3cd0f5d4bdeeaa8f968-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal\" (UID: \"4d1e80e0bdfbb3cd0f5d4bdeeaa8f968\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.101280 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.101257    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6aac1761331084f3b830a859c99fcd9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-251.ec2.internal\" (UID: \"d6aac1761331084f3b830a859c99fcd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.101342 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.101281    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6aac1761331084f3b830a859c99fcd9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-251.ec2.internal\" (UID: \"d6aac1761331084f3b830a859c99fcd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.101342 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.101302    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d1e80e0bdfbb3cd0f5d4bdeeaa8f968-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal\" (UID: \"4d1e80e0bdfbb3cd0f5d4bdeeaa8f968\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.101342 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.101323    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d1e80e0bdfbb3cd0f5d4bdeeaa8f968-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal\" (UID: \"4d1e80e0bdfbb3cd0f5d4bdeeaa8f968\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.165305 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.165280    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.169992 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.169977    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.175792 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.175766    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:07.276308 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.276283    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:07.377610 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.377585    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:07.479527 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.479501    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-251.ec2.internal\" not found"
Sep 09 14:11:07.488976 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.488957    2532 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 14:11:07.499171 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.499147    2532 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.513693 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.513676    2532 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 14:11:07.515473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.515457    2532 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 14:11:07.515633 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.515619    2532 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal"
Sep 09 14:11:07.521951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.521938    2532 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 14:11:07.541516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.541496    2532 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 14:11:07.541634 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.541614    2532 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 14:11:07.541679 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.541650    2532 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 14:11:07.541679 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.541665    2532 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 14:11:07.580153 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.580129    2532 apiserver.go:52] "Watching apiserver"
Sep 09 14:11:07.598815 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.598788    2532 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 14:11:07.599665 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.599616    2532 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 14:11:07.600031 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.600010    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-dghxg","openshift-multus/network-metrics-daemon-lq49r","openshift-network-operator/iptables-alerter-bgmdl","kube-system/konnectivity-agent-svpv6","openshift-cluster-node-tuning-operator/tuned-p5tl6","openshift-image-registry/node-ca-4z47c","openshift-multus/multus-additional-cni-plugins-t5cc9","openshift-network-diagnostics/network-check-target-pqw2f","openshift-ovn-kubernetes/ovnkube-node-pxnxs","kube-system/global-pull-secret-syncer-8xz8x","kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw","openshift-dns/node-resolver-cn577","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal"]
Sep 09 14:11:07.601896 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.601881    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.603010 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.602967    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.604297 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.604281    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.604983 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.604964    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r59z9\""
Sep 09 14:11:07.605049 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.604967    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 14:11:07.605332 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.605316    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 14:11:07.605451 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.605388    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 14:11:07.605546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.605531    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:07.605629 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.605612    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 14:11:07.605832 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.605809    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 14:11:07.605832 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.605819    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7spm4\""
Sep 09 14:11:07.605956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.605940    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 14:11:07.605996 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.605943    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 14:11:07.606656 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.606640    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 14:11:07.606740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.606675    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 14:11:07.606740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.606706    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 14:11:07.606871 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.606856    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.607707 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.607691    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z9f9m\""
Sep 09 14:11:07.608128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.608112    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 14:11:07.608194 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.608152    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:07.608357 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.608341    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2lbkx\""
Sep 09 14:11:07.608417 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.608361    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 14:11:07.608784 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.608757    2532 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 14:06:07 +0000 UTC" deadline="2028-04-21 15:30:03.320759744 +0000 UTC"
Sep 09 14:11:07.608840 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.608784    2532 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14161h18m55.711979361s"
Sep 09 14:11:07.609388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.609363    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 14:11:07.609388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.609373    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5shc7\""
Sep 09 14:11:07.609537 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.609397    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 14:11:07.609537 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.609522    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.610557 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.610539    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 14:11:07.610833 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.610815    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dbfvh\""
Sep 09 14:11:07.610929 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.610836    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 14:11:07.610929 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.610921    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 14:11:07.611042 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.610982    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.612134 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.612111    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 14:11:07.612240 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.612159    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 14:11:07.612304 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.612285    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 14:11:07.612367 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.612355    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h4k89\""
Sep 09 14:11:07.612420 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.612366    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:07.612473 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.612427    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:07.612549 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.612532    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 14:11:07.612549 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.612543    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 14:11:07.612849 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.612832    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 14:11:07.613282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.613264    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-296hk\""
Sep 09 14:11:07.613282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.613278    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 14:11:07.613772 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.613755    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:07.613844 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.613807    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:07.614404 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.614385    2532 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 14:11:07.615132 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.615116    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:07.615221 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.615179    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:07.616489 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.616470    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.619025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.619006    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 14:11:07.619200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.619064    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fwm9w\""
Sep 09 14:11:07.619200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.619143    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 14:11:07.619311 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.619297    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 14:11:07.642286 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.642270    2532 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 14:11:07.642457 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.642439    2532 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d9kkt"
Sep 09 14:11:07.653119 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.653092    2532 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d9kkt"
Sep 09 14:11:07.700034 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.700018    2532 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 14:11:07.705335 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705312    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmzn\" (UniqueName: \"kubernetes.io/projected/bd746049-41ca-4402-93a6-320c0828fcbb-kube-api-access-gtmzn\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.705413 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705351    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-sysconfig\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.705413 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705382    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-lib-modules\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.705413 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705407    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-tuned\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.705560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705430    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-registration-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.705560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705458    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-multus-cni-dir\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.705560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705490    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-run-netns\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.705560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705506    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-var-lib-kubelet\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.705560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705529    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-etc-kubernetes\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.705560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705553    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-sysctl-conf\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.705793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705581    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.705793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705609    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-device-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.705793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705636    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-etc-selinux\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.705793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705664    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-run-ovn-kubernetes\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.705793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705688    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-multus-conf-dir\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.705793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705717    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqjwf\" (UniqueName: \"kubernetes.io/projected/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-kube-api-access-nqjwf\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.705793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705743    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/49ea2ecc-a574-4969-85d4-1c6419ce4a06-konnectivity-ca\") pod \"konnectivity-agent-svpv6\" (UID: \"49ea2ecc-a574-4969-85d4-1c6419ce4a06\") " pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:07.705793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705774    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-etc-openvswitch\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705791    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8fae84b9-0f9a-4162-84b2-f3488a0e04af-env-overrides\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705830    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705853    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-modprobe-d\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705875    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-cni-bin\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705901    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705935    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfpqx\" (UniqueName: \"kubernetes.io/projected/8fae84b9-0f9a-4162-84b2-f3488a0e04af-kube-api-access-vfpqx\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705963    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-var-lib-cni-bin\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.705987    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64100743-6529-42aa-8204-8138f81c29c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706011    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-var-lib-kubelet\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706051    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04d39c37-e950-4def-86cf-bc268ada8e3f-dbus\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:07.706089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706086    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64100743-6529-42aa-8204-8138f81c29c7-system-cni-dir\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706134    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64100743-6529-42aa-8204-8138f81c29c7-cnibin\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706169    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be545f7f-c99c-47c4-9e0e-1e690afab968-tmp-dir\") pod \"node-resolver-cn577\" (UID: \"be545f7f-c99c-47c4-9e0e-1e690afab968\") " pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706209    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5044a9fc-f558-40d3-9525-f07cb8171534-host\") pod \"node-ca-4z47c\" (UID: \"5044a9fc-f558-40d3-9525-f07cb8171534\") " pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706261    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-run-multus-certs\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706298    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04d39c37-e950-4def-86cf-bc268ada8e3f-kubelet-config\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706328    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-systemd\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706349    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd746049-41ca-4402-93a6-320c0828fcbb-multus-daemon-config\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706371    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706390    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-kubelet\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706411    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-run-ovn\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706430    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-log-socket\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706449    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-system-cni-dir\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706469    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-multus-socket-dir-parent\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706489    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-run-k8s-cni-cncf-io\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706509    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8fae84b9-0f9a-4162-84b2-f3488a0e04af-ovn-node-metrics-cert\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.706534 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706529    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/182b56a9-69f5-455e-a828-37ae40d559d4-iptables-alerter-script\") pod \"iptables-alerter-bgmdl\" (UID: \"182b56a9-69f5-455e-a828-37ae40d559d4\") " pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706550    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9lq\" (UniqueName: \"kubernetes.io/projected/182b56a9-69f5-455e-a828-37ae40d559d4-kube-api-access-dx9lq\") pod \"iptables-alerter-bgmdl\" (UID: \"182b56a9-69f5-455e-a828-37ae40d559d4\") " pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706569    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-kubernetes\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706589    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-run\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706609    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-sys-fs\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706630    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-host\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706650    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbvq\" (UniqueName: \"kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq\") pod \"network-check-target-pqw2f\" (UID: \"be00a0fe-fb34-4722-be6e-6ac18ccaed54\") " pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706688    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnlv5\" (UniqueName: \"kubernetes.io/projected/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-kube-api-access-fnlv5\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706712    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwgjs\" (UniqueName: \"kubernetes.io/projected/5044a9fc-f558-40d3-9525-f07cb8171534-kube-api-access-nwgjs\") pod \"node-ca-4z47c\" (UID: \"5044a9fc-f558-40d3-9525-f07cb8171534\") " pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706730    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-run-openvswitch\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706758    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-hostroot\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706783    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjcf\" (UniqueName: \"kubernetes.io/projected/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-kube-api-access-zkjcf\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706800    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-sys\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706832    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-tmp\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706853    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be545f7f-c99c-47c4-9e0e-1e690afab968-hosts-file\") pod \"node-resolver-cn577\" (UID: \"be545f7f-c99c-47c4-9e0e-1e690afab968\") " pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706868    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwclg\" (UniqueName: \"kubernetes.io/projected/be545f7f-c99c-47c4-9e0e-1e690afab968-kube-api-access-fwclg\") pod \"node-resolver-cn577\" (UID: \"be545f7f-c99c-47c4-9e0e-1e690afab968\") " pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.707148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706898    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd746049-41ca-4402-93a6-320c0828fcbb-cni-binary-copy\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706920    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5044a9fc-f558-40d3-9525-f07cb8171534-serviceca\") pod \"node-ca-4z47c\" (UID: \"5044a9fc-f558-40d3-9525-f07cb8171534\") " pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706949    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8fae84b9-0f9a-4162-84b2-f3488a0e04af-ovnkube-script-lib\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.706982    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64100743-6529-42aa-8204-8138f81c29c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707030    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjx8\" (UniqueName: \"kubernetes.io/projected/64100743-6529-42aa-8204-8138f81c29c7-kube-api-access-5qjx8\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707068    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/182b56a9-69f5-455e-a828-37ae40d559d4-host-slash\") pod \"iptables-alerter-bgmdl\" (UID: \"182b56a9-69f5-455e-a828-37ae40d559d4\") " pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707107    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-slash\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707152    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-sysctl-d\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707176    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-var-lib-openvswitch\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707199    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-node-log\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707224    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8fae84b9-0f9a-4162-84b2-f3488a0e04af-ovnkube-config\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707255    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64100743-6529-42aa-8204-8138f81c29c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707284    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-run-netns\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707309    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-run-systemd\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707332    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-socket-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707355    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/49ea2ecc-a574-4969-85d4-1c6419ce4a06-agent-certs\") pod \"konnectivity-agent-svpv6\" (UID: \"49ea2ecc-a574-4969-85d4-1c6419ce4a06\") " pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:07.707743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707387    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-systemd-units\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.708540 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707420    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-cnibin\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.708540 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707459    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-var-lib-cni-multus\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.708540 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707499    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64100743-6529-42aa-8204-8138f81c29c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.708540 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.707535    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-cni-netd\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.808744 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.808708    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8fae84b9-0f9a-4162-84b2-f3488a0e04af-ovnkube-config\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.808925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.808755    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64100743-6529-42aa-8204-8138f81c29c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.808925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.808803    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-run-netns\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.808925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.808828    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-run-systemd\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.808925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.808853    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-socket-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.808925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.808881    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/49ea2ecc-a574-4969-85d4-1c6419ce4a06-agent-certs\") pod \"konnectivity-agent-svpv6\" (UID: \"49ea2ecc-a574-4969-85d4-1c6419ce4a06\") " pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:07.808925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.808912    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-run-netns\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.808917    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-run-systemd\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.808962    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-systemd-units\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.808994    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-cnibin\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809015    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-socket-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809021    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-var-lib-cni-multus\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809061    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-var-lib-cni-multus\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809086    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64100743-6529-42aa-8204-8138f81c29c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809114    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-cni-netd\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809141    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmzn\" (UniqueName: \"kubernetes.io/projected/bd746049-41ca-4402-93a6-320c0828fcbb-kube-api-access-gtmzn\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809144    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-systemd-units\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809142    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-cnibin\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809182    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-cni-netd\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.809229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809217    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-sysconfig\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.809783 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809331    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-lib-modules\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.809783 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809643    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-tuned\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.809783 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809684    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-registration-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.809783 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809730    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-multus-cni-dir\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.809783 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809761    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-run-netns\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809786    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-var-lib-kubelet\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809817    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-etc-kubernetes\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809849    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-sysctl-conf\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809859    2532 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809881    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809913    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-device-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809928    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64100743-6529-42aa-8204-8138f81c29c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809936    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-etc-selinux\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809968    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-run-ovn-kubernetes\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.809997    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-multus-conf-dir\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.810025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.810029    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqjwf\" (UniqueName: \"kubernetes.io/projected/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-kube-api-access-nqjwf\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.810670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.810061    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/49ea2ecc-a574-4969-85d4-1c6419ce4a06-konnectivity-ca\") pod \"konnectivity-agent-svpv6\" (UID: \"49ea2ecc-a574-4969-85d4-1c6419ce4a06\") " pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:07.810670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.810090    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8fae84b9-0f9a-4162-84b2-f3488a0e04af-ovnkube-config\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.810670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.810130    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-run-netns\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.810670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.810318    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64100743-6529-42aa-8204-8138f81c29c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.810878 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.810750    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/49ea2ecc-a574-4969-85d4-1c6419ce4a06-konnectivity-ca\") pod \"konnectivity-agent-svpv6\" (UID: \"49ea2ecc-a574-4969-85d4-1c6419ce4a06\") " pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:07.810878 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.810852    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.810979 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.810870    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-etc-selinux\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.810979 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.810936    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-multus-conf-dir\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.811080 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811001    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-multus-cni-dir\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.811131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811080    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-lib-modules\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.811294 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811229    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-sysconfig\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.811294 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811258    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-etc-kubernetes\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.811424 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811294    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-registration-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.811424 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811317    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-run-ovn-kubernetes\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.811424 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811367    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-device-dir\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.811424 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811395    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-var-lib-kubelet\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.811614 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811468    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-sysctl-conf\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.811614 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811526    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-etc-openvswitch\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.811614 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811567    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8fae84b9-0f9a-4162-84b2-f3488a0e04af-env-overrides\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.811614 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811598    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:07.811614 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811603    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-etc-openvswitch\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.811859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811631    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-modprobe-d\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.811859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811682    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-cni-bin\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.811859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811713    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.811859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811750    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfpqx\" (UniqueName: \"kubernetes.io/projected/8fae84b9-0f9a-4162-84b2-f3488a0e04af-kube-api-access-vfpqx\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.811859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811774    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-var-lib-cni-bin\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.811859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811802    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64100743-6529-42aa-8204-8138f81c29c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.811859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811834    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-var-lib-kubelet\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811866    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04d39c37-e950-4def-86cf-bc268ada8e3f-dbus\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811897    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64100743-6529-42aa-8204-8138f81c29c7-system-cni-dir\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811931    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64100743-6529-42aa-8204-8138f81c29c7-cnibin\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.811967    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be545f7f-c99c-47c4-9e0e-1e690afab968-tmp-dir\") pod \"node-resolver-cn577\" (UID: \"be545f7f-c99c-47c4-9e0e-1e690afab968\") " pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812005    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5044a9fc-f558-40d3-9525-f07cb8171534-host\") pod \"node-ca-4z47c\" (UID: \"5044a9fc-f558-40d3-9525-f07cb8171534\") " pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812042    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-run-multus-certs\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812075    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04d39c37-e950-4def-86cf-bc268ada8e3f-kubelet-config\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812138    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-systemd\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812175    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd746049-41ca-4402-93a6-320c0828fcbb-multus-daemon-config\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812210    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:07.812249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812247    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-kubelet\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812275    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-run-ovn\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812309    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-log-socket\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812341    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-system-cni-dir\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812373    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-multus-socket-dir-parent\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812401    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-run-k8s-cni-cncf-io\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812406    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-cni-bin\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812432    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8fae84b9-0f9a-4162-84b2-f3488a0e04af-ovn-node-metrics-cert\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812467    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/182b56a9-69f5-455e-a828-37ae40d559d4-iptables-alerter-script\") pod \"iptables-alerter-bgmdl\" (UID: \"182b56a9-69f5-455e-a828-37ae40d559d4\") " pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812504    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9lq\" (UniqueName: \"kubernetes.io/projected/182b56a9-69f5-455e-a828-37ae40d559d4-kube-api-access-dx9lq\") pod \"iptables-alerter-bgmdl\" (UID: \"182b56a9-69f5-455e-a828-37ae40d559d4\") " pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.812524    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812538    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-kubernetes\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812567    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-run\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.812624    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret podName:04d39c37-e950-4def-86cf-bc268ada8e3f nodeName:}" failed. No retries permitted until 2026-09-09 14:11:08.31258393 +0000 UTC m=+2.075379166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret") pod "global-pull-secret-syncer-8xz8x" (UID: "04d39c37-e950-4def-86cf-bc268ada8e3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812645    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-sys-fs\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812651    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-run\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812684    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-host\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.812807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812717    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-var-lib-kubelet\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812728    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbvq\" (UniqueName: \"kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq\") pod \"network-check-target-pqw2f\" (UID: \"be00a0fe-fb34-4722-be6e-6ac18ccaed54\") " pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812763    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnlv5\" (UniqueName: \"kubernetes.io/projected/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-kube-api-access-fnlv5\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812779    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04d39c37-e950-4def-86cf-bc268ada8e3f-dbus\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812796    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwgjs\" (UniqueName: \"kubernetes.io/projected/5044a9fc-f558-40d3-9525-f07cb8171534-kube-api-access-nwgjs\") pod \"node-ca-4z47c\" (UID: \"5044a9fc-f558-40d3-9525-f07cb8171534\") " pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812862    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64100743-6529-42aa-8204-8138f81c29c7-cnibin\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812919    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-modprobe-d\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812923    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64100743-6529-42aa-8204-8138f81c29c7-system-cni-dir\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812948    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813040    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-var-lib-cni-bin\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813098    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-kubelet\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813140    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-run-ovn\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813234    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-run-openvswitch\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813256    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-log-socket\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813273    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-hostroot\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813307    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjcf\" (UniqueName: \"kubernetes.io/projected/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-kube-api-access-zkjcf\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:07.813680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813332    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-sys\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814037    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-kubernetes\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813455    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-multus-socket-dir-parent\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813455    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be545f7f-c99c-47c4-9e0e-1e690afab968-tmp-dir\") pod \"node-resolver-cn577\" (UID: \"be545f7f-c99c-47c4-9e0e-1e690afab968\") " pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813512    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5044a9fc-f558-40d3-9525-f07cb8171534-host\") pod \"node-ca-4z47c\" (UID: \"5044a9fc-f558-40d3-9525-f07cb8171534\") " pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.812006    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8fae84b9-0f9a-4162-84b2-f3488a0e04af-env-overrides\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814300    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-run-k8s-cni-cncf-io\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814322    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-tmp\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.813366    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-system-cni-dir\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814405    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be545f7f-c99c-47c4-9e0e-1e690afab968-hosts-file\") pod \"node-resolver-cn577\" (UID: \"be545f7f-c99c-47c4-9e0e-1e690afab968\") " pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814448    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwclg\" (UniqueName: \"kubernetes.io/projected/be545f7f-c99c-47c4-9e0e-1e690afab968-kube-api-access-fwclg\") pod \"node-resolver-cn577\" (UID: \"be545f7f-c99c-47c4-9e0e-1e690afab968\") " pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814495    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd746049-41ca-4402-93a6-320c0828fcbb-cni-binary-copy\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.814546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814526    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5044a9fc-f558-40d3-9525-f07cb8171534-serviceca\") pod \"node-ca-4z47c\" (UID: \"5044a9fc-f558-40d3-9525-f07cb8171534\") " pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:07.815128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814559    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8fae84b9-0f9a-4162-84b2-f3488a0e04af-ovnkube-script-lib\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.815128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814586    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64100743-6529-42aa-8204-8138f81c29c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.815128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814702    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjx8\" (UniqueName: \"kubernetes.io/projected/64100743-6529-42aa-8204-8138f81c29c7-kube-api-access-5qjx8\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.815128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814748    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/182b56a9-69f5-455e-a828-37ae40d559d4-host-slash\") pod \"iptables-alerter-bgmdl\" (UID: \"182b56a9-69f5-455e-a828-37ae40d559d4\") " pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.815128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814751    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-tuned\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.815128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814780    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-slash\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.815128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814805    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-sysctl-d\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.815128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814851    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-var-lib-openvswitch\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.815128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814909    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-node-log\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.815128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.815089    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-node-log\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.815695 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.815194    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-host-run-multus-certs\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.815695 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.815256    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04d39c37-e950-4def-86cf-bc268ada8e3f-kubelet-config\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:07.815695 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.815342    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-systemd\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.815695 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.815377    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/49ea2ecc-a574-4969-85d4-1c6419ce4a06-agent-certs\") pod \"konnectivity-agent-svpv6\" (UID: \"49ea2ecc-a574-4969-85d4-1c6419ce4a06\") " pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:07.815695 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.815515    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:07.815695 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.815640    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs podName:9a8d7ec4-9e2a-40c5-b008-3c6890b621fe nodeName:}" failed. No retries permitted until 2026-09-09 14:11:08.315620091 +0000 UTC m=+2.078415318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs") pod "network-metrics-daemon-lq49r" (UID: "9a8d7ec4-9e2a-40c5-b008-3c6890b621fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:07.816013 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.815719    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-sys\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.816013 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.814539    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64100743-6529-42aa-8204-8138f81c29c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.816013 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.815862    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-run-openvswitch\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.816013 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.815988    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-sys-fs\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.816013 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.815996    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64100743-6529-42aa-8204-8138f81c29c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.816295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.816074    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd746049-41ca-4402-93a6-320c0828fcbb-hostroot\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.816295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.816105    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-etc-sysctl-d\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.816295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.816251    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd746049-41ca-4402-93a6-320c0828fcbb-multus-daemon-config\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.816444 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.816344    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-host\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.816444 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.816374    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-host-slash\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.816855 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.816674    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5044a9fc-f558-40d3-9525-f07cb8171534-serviceca\") pod \"node-ca-4z47c\" (UID: \"5044a9fc-f558-40d3-9525-f07cb8171534\") " pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:07.816855 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.816691    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd746049-41ca-4402-93a6-320c0828fcbb-cni-binary-copy\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.816855 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.816697    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8fae84b9-0f9a-4162-84b2-f3488a0e04af-ovn-node-metrics-cert\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.816855 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.816802    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/182b56a9-69f5-455e-a828-37ae40d559d4-host-slash\") pod \"iptables-alerter-bgmdl\" (UID: \"182b56a9-69f5-455e-a828-37ae40d559d4\") " pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.817104 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.816814    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be545f7f-c99c-47c4-9e0e-1e690afab968-hosts-file\") pod \"node-resolver-cn577\" (UID: \"be545f7f-c99c-47c4-9e0e-1e690afab968\") " pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.817104 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.817023    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8fae84b9-0f9a-4162-84b2-f3488a0e04af-var-lib-openvswitch\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.817360 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.817325    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8fae84b9-0f9a-4162-84b2-f3488a0e04af-ovnkube-script-lib\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.817591 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.817575    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/182b56a9-69f5-455e-a828-37ae40d559d4-iptables-alerter-script\") pod \"iptables-alerter-bgmdl\" (UID: \"182b56a9-69f5-455e-a828-37ae40d559d4\") " pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.818417 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.818373    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-tmp\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.819555 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.819524    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmzn\" (UniqueName: \"kubernetes.io/projected/bd746049-41ca-4402-93a6-320c0828fcbb-kube-api-access-gtmzn\") pod \"multus-dghxg\" (UID: \"bd746049-41ca-4402-93a6-320c0828fcbb\") " pod="openshift-multus/multus-dghxg"
Sep 09 14:11:07.819655 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.819594    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqjwf\" (UniqueName: \"kubernetes.io/projected/d66b4e8e-2f59-49c1-a992-c3e45fc004a7-kube-api-access-nqjwf\") pod \"tuned-p5tl6\" (UID: \"d66b4e8e-2f59-49c1-a992-c3e45fc004a7\") " pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:07.823227 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.823205    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 14:11:07.823328 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.823235    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 14:11:07.823328 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.823249    2532 projected.go:196] Error preparing data for projected volume kube-api-access-gkbvq for pod openshift-network-diagnostics/network-check-target-pqw2f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:07.823328 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:07.823305    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq podName:be00a0fe-fb34-4722-be6e-6ac18ccaed54 nodeName:}" failed. No retries permitted until 2026-09-09 14:11:08.323287603 +0000 UTC m=+2.086082825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gkbvq" (UniqueName: "kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq") pod "network-check-target-pqw2f" (UID: "be00a0fe-fb34-4722-be6e-6ac18ccaed54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:07.823573 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.823546    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwgjs\" (UniqueName: \"kubernetes.io/projected/5044a9fc-f558-40d3-9525-f07cb8171534-kube-api-access-nwgjs\") pod \"node-ca-4z47c\" (UID: \"5044a9fc-f558-40d3-9525-f07cb8171534\") " pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:07.824272 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.824247    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfpqx\" (UniqueName: \"kubernetes.io/projected/8fae84b9-0f9a-4162-84b2-f3488a0e04af-kube-api-access-vfpqx\") pod \"ovnkube-node-pxnxs\" (UID: \"8fae84b9-0f9a-4162-84b2-f3488a0e04af\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:07.824993 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.824961    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwclg\" (UniqueName: \"kubernetes.io/projected/be545f7f-c99c-47c4-9e0e-1e690afab968-kube-api-access-fwclg\") pod \"node-resolver-cn577\" (UID: \"be545f7f-c99c-47c4-9e0e-1e690afab968\") " pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.825214 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.825195    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9lq\" (UniqueName: \"kubernetes.io/projected/182b56a9-69f5-455e-a828-37ae40d559d4-kube-api-access-dx9lq\") pod \"iptables-alerter-bgmdl\" (UID: \"182b56a9-69f5-455e-a828-37ae40d559d4\") " pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.825504 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.825486    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnlv5\" (UniqueName: \"kubernetes.io/projected/154c326a-a75e-49ea-a5f2-6fa03c5cbe55-kube-api-access-fnlv5\") pod \"aws-ebs-csi-driver-node-hm7xw\" (UID: \"154c326a-a75e-49ea-a5f2-6fa03c5cbe55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:07.825504 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.825494    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjx8\" (UniqueName: \"kubernetes.io/projected/64100743-6529-42aa-8204-8138f81c29c7-kube-api-access-5qjx8\") pod \"multus-additional-cni-plugins-t5cc9\" (UID: \"64100743-6529-42aa-8204-8138f81c29c7\") " pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.825628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.825611    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjcf\" (UniqueName: \"kubernetes.io/projected/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-kube-api-access-zkjcf\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:07.869052 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:07.869017    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6aac1761331084f3b830a859c99fcd9.slice/crio-5ec761992dc9d8d2e46962560e95fd7d45112c89c6af7b0a6559289898191812 WatchSource:0}: Error finding container 5ec761992dc9d8d2e46962560e95fd7d45112c89c6af7b0a6559289898191812: Status 404 returned error can't find the container with id 5ec761992dc9d8d2e46962560e95fd7d45112c89c6af7b0a6559289898191812
Sep 09 14:11:07.869299 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:07.869279    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1e80e0bdfbb3cd0f5d4bdeeaa8f968.slice/crio-3f7de213bb6e4b80c6524717722a56dc303a3dea6c5d82d5ed142ba0851b5dbf WatchSource:0}: Error finding container 3f7de213bb6e4b80c6524717722a56dc303a3dea6c5d82d5ed142ba0851b5dbf: Status 404 returned error can't find the container with id 3f7de213bb6e4b80c6524717722a56dc303a3dea6c5d82d5ed142ba0851b5dbf
Sep 09 14:11:07.874543 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.874522    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 14:11:07.940615 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.940591    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t5cc9"
Sep 09 14:11:07.948547 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:07.948527    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64100743_6529_42aa_8204_8138f81c29c7.slice/crio-3a1243f549be418f1ea2da4ffa016b1a92d76f8101501be2c54c7de67f4a1cdb WatchSource:0}: Error finding container 3a1243f549be418f1ea2da4ffa016b1a92d76f8101501be2c54c7de67f4a1cdb: Status 404 returned error can't find the container with id 3a1243f549be418f1ea2da4ffa016b1a92d76f8101501be2c54c7de67f4a1cdb
Sep 09 14:11:07.954479 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.954460    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cn577"
Sep 09 14:11:07.961648 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:07.961624    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe545f7f_c99c_47c4_9e0e_1e690afab968.slice/crio-49b36797fba6a6ba4efe8c53c5ef2008b26cd423764e99b433681e466ad42757 WatchSource:0}: Error finding container 49b36797fba6a6ba4efe8c53c5ef2008b26cd423764e99b433681e466ad42757: Status 404 returned error can't find the container with id 49b36797fba6a6ba4efe8c53c5ef2008b26cd423764e99b433681e466ad42757
Sep 09 14:11:07.974726 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.974711    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bgmdl"
Sep 09 14:11:07.980989 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:07.980969    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182b56a9_69f5_455e_a828_37ae40d559d4.slice/crio-49ab5ab2f1b689811a4a62acb3ba9caa1828f3d0be5f0f05028e7e2aace99ce4 WatchSource:0}: Error finding container 49ab5ab2f1b689811a4a62acb3ba9caa1828f3d0be5f0f05028e7e2aace99ce4: Status 404 returned error can't find the container with id 49ab5ab2f1b689811a4a62acb3ba9caa1828f3d0be5f0f05028e7e2aace99ce4
Sep 09 14:11:07.991407 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:07.991388    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:07.997826 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:07.997808    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea2ecc_a574_4969_85d4_1c6419ce4a06.slice/crio-85e420c3aba687c97d00d3a5d243e2d0ee4565f4e4ac3e49383c7246911e021c WatchSource:0}: Error finding container 85e420c3aba687c97d00d3a5d243e2d0ee4565f4e4ac3e49383c7246911e021c: Status 404 returned error can't find the container with id 85e420c3aba687c97d00d3a5d243e2d0ee4565f4e4ac3e49383c7246911e021c
Sep 09 14:11:08.008839 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.008822    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p5tl6"
Sep 09 14:11:08.015388 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:08.015366    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66b4e8e_2f59_49c1_a992_c3e45fc004a7.slice/crio-72fa6c60d4275884ef66548f900f799a0547b57f8d679e79d06a647f71880002 WatchSource:0}: Error finding container 72fa6c60d4275884ef66548f900f799a0547b57f8d679e79d06a647f71880002: Status 404 returned error can't find the container with id 72fa6c60d4275884ef66548f900f799a0547b57f8d679e79d06a647f71880002
Sep 09 14:11:08.015965 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.015943    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4z47c"
Sep 09 14:11:08.022671 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:08.022649    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5044a9fc_f558_40d3_9525_f07cb8171534.slice/crio-b59ae467c4994b0c9e6f9cd027b966aef112cb1fba159620d6d004f97b6d631c WatchSource:0}: Error finding container b59ae467c4994b0c9e6f9cd027b966aef112cb1fba159620d6d004f97b6d631c: Status 404 returned error can't find the container with id b59ae467c4994b0c9e6f9cd027b966aef112cb1fba159620d6d004f97b6d631c
Sep 09 14:11:08.037129 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.037101    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:08.044190 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:08.044153    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fae84b9_0f9a_4162_84b2_f3488a0e04af.slice/crio-7d2d50fd14836aaf0eba99cc7d46b1f2bcd6e72283bc41022bc33aa684150432 WatchSource:0}: Error finding container 7d2d50fd14836aaf0eba99cc7d46b1f2bcd6e72283bc41022bc33aa684150432: Status 404 returned error can't find the container with id 7d2d50fd14836aaf0eba99cc7d46b1f2bcd6e72283bc41022bc33aa684150432
Sep 09 14:11:08.052261 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.052245    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dghxg"
Sep 09 14:11:08.058285 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.058263    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw"
Sep 09 14:11:08.058897 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:08.058877    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd746049_41ca_4402_93a6_320c0828fcbb.slice/crio-3ae45ad151f1713091dd36a2bedcd36c2fba73263d6e97bb938d9520b59872ad WatchSource:0}: Error finding container 3ae45ad151f1713091dd36a2bedcd36c2fba73263d6e97bb938d9520b59872ad: Status 404 returned error can't find the container with id 3ae45ad151f1713091dd36a2bedcd36c2fba73263d6e97bb938d9520b59872ad
Sep 09 14:11:08.065533 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:08.065513    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154c326a_a75e_49ea_a5f2_6fa03c5cbe55.slice/crio-fa681c34e99853a117efe8dc5f483f03f8fab77514f02645ed01d5a91660acc2 WatchSource:0}: Error finding container fa681c34e99853a117efe8dc5f483f03f8fab77514f02645ed01d5a91660acc2: Status 404 returned error can't find the container with id fa681c34e99853a117efe8dc5f483f03f8fab77514f02645ed01d5a91660acc2
Sep 09 14:11:08.067436 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.067416    2532 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-driver start failed in pod aws-ebs-csi-driver-node-hm7xw_openshift-cluster-csi-drivers(154c326a-a75e-49ea-a5f2-6fa03c5cbe55): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 09 14:11:08.068122 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.068104    2532 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-node-driver-registrar start failed in pod aws-ebs-csi-driver-node-hm7xw_openshift-cluster-csi-drivers(154c326a-a75e-49ea-a5f2-6fa03c5cbe55): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 09 14:11:08.069700 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.069683    2532 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-liveness-probe start failed in pod aws-ebs-csi-driver-node-hm7xw_openshift-cluster-csi-drivers(154c326a-a75e-49ea-a5f2-6fa03c5cbe55): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 09 14:11:08.069751 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.069732    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ErrImagePull: \"pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw" podUID="154c326a-a75e-49ea-a5f2-6fa03c5cbe55"
Sep 09 14:11:08.318223 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.318152    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:08.318223 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.318199    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:08.318379 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.318295    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:08.318379 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.318302    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:08.318379 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.318344    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret podName:04d39c37-e950-4def-86cf-bc268ada8e3f nodeName:}" failed. No retries permitted until 2026-09-09 14:11:09.318330073 +0000 UTC m=+3.081125298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret") pod "global-pull-secret-syncer-8xz8x" (UID: "04d39c37-e950-4def-86cf-bc268ada8e3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:08.318379 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.318360    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs podName:9a8d7ec4-9e2a-40c5-b008-3c6890b621fe nodeName:}" failed. No retries permitted until 2026-09-09 14:11:09.31835271 +0000 UTC m=+3.081147929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs") pod "network-metrics-daemon-lq49r" (UID: "9a8d7ec4-9e2a-40c5-b008-3c6890b621fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:08.418826 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.418800    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbvq\" (UniqueName: \"kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq\") pod \"network-check-target-pqw2f\" (UID: \"be00a0fe-fb34-4722-be6e-6ac18ccaed54\") " pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:08.418976 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.418938    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 14:11:08.418976 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.418952    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 14:11:08.418976 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.418964    2532 projected.go:196] Error preparing data for projected volume kube-api-access-gkbvq for pod openshift-network-diagnostics/network-check-target-pqw2f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:08.419163 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.419009    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq podName:be00a0fe-fb34-4722-be6e-6ac18ccaed54 nodeName:}" failed. No retries permitted until 2026-09-09 14:11:09.418993899 +0000 UTC m=+3.181789118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkbvq" (UniqueName: "kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq") pod "network-check-target-pqw2f" (UID: "be00a0fe-fb34-4722-be6e-6ac18ccaed54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:08.655659 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.655563    2532 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 14:06:08 +0000 UTC" deadline="2028-06-12 16:37:04.51353596 +0000 UTC"
Sep 09 14:11:08.655659 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.655612    2532 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15410h25m55.857928283s"
Sep 09 14:11:08.732801 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.728884    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:08.732801 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.729093    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:08.735852 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.735745    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5cc9" event={"ID":"64100743-6529-42aa-8204-8138f81c29c7","Type":"ContainerStarted","Data":"3a1243f549be418f1ea2da4ffa016b1a92d76f8101501be2c54c7de67f4a1cdb"}
Sep 09 14:11:08.738094 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.738066    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal" event={"ID":"d6aac1761331084f3b830a859c99fcd9","Type":"ContainerStarted","Data":"5ec761992dc9d8d2e46962560e95fd7d45112c89c6af7b0a6559289898191812"}
Sep 09 14:11:08.740625 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.740574    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal" event={"ID":"4d1e80e0bdfbb3cd0f5d4bdeeaa8f968","Type":"ContainerStarted","Data":"3f7de213bb6e4b80c6524717722a56dc303a3dea6c5d82d5ed142ba0851b5dbf"}
Sep 09 14:11:08.742261 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.742236    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw" event={"ID":"154c326a-a75e-49ea-a5f2-6fa03c5cbe55","Type":"ContainerStarted","Data":"fa681c34e99853a117efe8dc5f483f03f8fab77514f02645ed01d5a91660acc2"}
Sep 09 14:11:08.748989 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.748962    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dghxg" event={"ID":"bd746049-41ca-4402-93a6-320c0828fcbb","Type":"ContainerStarted","Data":"3ae45ad151f1713091dd36a2bedcd36c2fba73263d6e97bb938d9520b59872ad"}
Sep 09 14:11:08.750875 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:08.750834    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:442f6f711db8a6f8fae4dfff48a29e5fb0d154b2c6779c5e2e46d649f99ccfec\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:909d94acdf16bd26857849698608cb17db0c62ca3eab7d670ee858a6e839390e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e3a2f3c9f8d7a459ccd0719643fba2d050d158d387ade828c3d58813bc3b6e4\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw" podUID="154c326a-a75e-49ea-a5f2-6fa03c5cbe55"
Sep 09 14:11:08.751043 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.751022    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" event={"ID":"8fae84b9-0f9a-4162-84b2-f3488a0e04af","Type":"ContainerStarted","Data":"7d2d50fd14836aaf0eba99cc7d46b1f2bcd6e72283bc41022bc33aa684150432"}
Sep 09 14:11:08.757144 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.757119    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4z47c" event={"ID":"5044a9fc-f558-40d3-9525-f07cb8171534","Type":"ContainerStarted","Data":"b59ae467c4994b0c9e6f9cd027b966aef112cb1fba159620d6d004f97b6d631c"}
Sep 09 14:11:08.762102 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.760968    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-svpv6" event={"ID":"49ea2ecc-a574-4969-85d4-1c6419ce4a06","Type":"ContainerStarted","Data":"85e420c3aba687c97d00d3a5d243e2d0ee4565f4e4ac3e49383c7246911e021c"}
Sep 09 14:11:08.763821 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.763794    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bgmdl" event={"ID":"182b56a9-69f5-455e-a828-37ae40d559d4","Type":"ContainerStarted","Data":"49ab5ab2f1b689811a4a62acb3ba9caa1828f3d0be5f0f05028e7e2aace99ce4"}
Sep 09 14:11:08.768435 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.768409    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p5tl6" event={"ID":"d66b4e8e-2f59-49c1-a992-c3e45fc004a7","Type":"ContainerStarted","Data":"72fa6c60d4275884ef66548f900f799a0547b57f8d679e79d06a647f71880002"}
Sep 09 14:11:08.771569 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.771544    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cn577" event={"ID":"be545f7f-c99c-47c4-9e0e-1e690afab968","Type":"ContainerStarted","Data":"49b36797fba6a6ba4efe8c53c5ef2008b26cd423764e99b433681e466ad42757"}
Sep 09 14:11:08.991972 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:08.991809    2532 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 14:11:09.326184 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:09.326087    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:09.326184 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:09.326160    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:09.326433 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:09.326312    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:09.326433 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:09.326381    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs podName:9a8d7ec4-9e2a-40c5-b008-3c6890b621fe nodeName:}" failed. No retries permitted until 2026-09-09 14:11:11.326360642 +0000 UTC m=+5.089155887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs") pod "network-metrics-daemon-lq49r" (UID: "9a8d7ec4-9e2a-40c5-b008-3c6890b621fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:09.326840 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:09.326821    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:09.326926 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:09.326890    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret podName:04d39c37-e950-4def-86cf-bc268ada8e3f nodeName:}" failed. No retries permitted until 2026-09-09 14:11:11.326870192 +0000 UTC m=+5.089665412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret") pod "global-pull-secret-syncer-8xz8x" (UID: "04d39c37-e950-4def-86cf-bc268ada8e3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:09.427837 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:09.427179    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbvq\" (UniqueName: \"kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq\") pod \"network-check-target-pqw2f\" (UID: \"be00a0fe-fb34-4722-be6e-6ac18ccaed54\") " pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:09.427837 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:09.427378    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 14:11:09.427837 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:09.427402    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 14:11:09.427837 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:09.427416    2532 projected.go:196] Error preparing data for projected volume kube-api-access-gkbvq for pod openshift-network-diagnostics/network-check-target-pqw2f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:09.427837 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:09.427486    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq podName:be00a0fe-fb34-4722-be6e-6ac18ccaed54 nodeName:}" failed. No retries permitted until 2026-09-09 14:11:11.427466967 +0000 UTC m=+5.190262188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkbvq" (UniqueName: "kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq") pod "network-check-target-pqw2f" (UID: "be00a0fe-fb34-4722-be6e-6ac18ccaed54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:09.656739 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:09.656618    2532 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 14:06:08 +0000 UTC" deadline="2028-03-07 12:31:32.782548472 +0000 UTC"
Sep 09 14:11:09.656739 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:09.656689    2532 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13078h20m23.125864068s"
Sep 09 14:11:09.729016 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:09.728987    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:09.729213 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:09.729115    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:09.729302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:09.729230    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:09.729361 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:09.729314    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:10.143951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:10.143869    2532 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 14:11:10.352722 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:10.352650    2532 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 14:11:10.729373 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:10.729343    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:10.729951 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:10.729491    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:11.343655 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:11.343617    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:11.343851 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:11.343681    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:11.343851 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:11.343786    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:11.343851 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:11.343821    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:11.344020 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:11.343879    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs podName:9a8d7ec4-9e2a-40c5-b008-3c6890b621fe nodeName:}" failed. No retries permitted until 2026-09-09 14:11:15.34385918 +0000 UTC m=+9.106654402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs") pod "network-metrics-daemon-lq49r" (UID: "9a8d7ec4-9e2a-40c5-b008-3c6890b621fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:11.344020 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:11.343899    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret podName:04d39c37-e950-4def-86cf-bc268ada8e3f nodeName:}" failed. No retries permitted until 2026-09-09 14:11:15.343888445 +0000 UTC m=+9.106683670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret") pod "global-pull-secret-syncer-8xz8x" (UID: "04d39c37-e950-4def-86cf-bc268ada8e3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:11.445239 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:11.445156    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbvq\" (UniqueName: \"kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq\") pod \"network-check-target-pqw2f\" (UID: \"be00a0fe-fb34-4722-be6e-6ac18ccaed54\") " pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:11.445414 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:11.445294    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 14:11:11.445414 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:11.445317    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 14:11:11.445414 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:11.445330    2532 projected.go:196] Error preparing data for projected volume kube-api-access-gkbvq for pod openshift-network-diagnostics/network-check-target-pqw2f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:11.445414 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:11.445394    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq podName:be00a0fe-fb34-4722-be6e-6ac18ccaed54 nodeName:}" failed. No retries permitted until 2026-09-09 14:11:15.445376118 +0000 UTC m=+9.208171354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkbvq" (UniqueName: "kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq") pod "network-check-target-pqw2f" (UID: "be00a0fe-fb34-4722-be6e-6ac18ccaed54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:11.728846 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:11.728763    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:11.729008 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:11.728908    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:11.729329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:11.729309    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:11.729434 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:11.729410    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:12.728326 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:12.728281    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:12.728510 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:12.728442    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:13.728617 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:13.728526    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:13.729244 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:13.728674    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:13.729244 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:13.728534    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:13.729244 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:13.728839    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:14.729205 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:14.728349    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:14.729205 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:14.728506    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:15.379585 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:15.379536    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:15.379776 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:15.379599    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:15.379776 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:15.379750    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:15.379890 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:15.379818    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs podName:9a8d7ec4-9e2a-40c5-b008-3c6890b621fe nodeName:}" failed. No retries permitted until 2026-09-09 14:11:23.379795648 +0000 UTC m=+17.142590873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs") pod "network-metrics-daemon-lq49r" (UID: "9a8d7ec4-9e2a-40c5-b008-3c6890b621fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:15.380081 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:15.380000    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:15.380081 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:15.380057    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret podName:04d39c37-e950-4def-86cf-bc268ada8e3f nodeName:}" failed. No retries permitted until 2026-09-09 14:11:23.38004229 +0000 UTC m=+17.142837514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret") pod "global-pull-secret-syncer-8xz8x" (UID: "04d39c37-e950-4def-86cf-bc268ada8e3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:15.480600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:15.480511    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbvq\" (UniqueName: \"kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq\") pod \"network-check-target-pqw2f\" (UID: \"be00a0fe-fb34-4722-be6e-6ac18ccaed54\") " pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:15.480792 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:15.480707    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 14:11:15.480792 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:15.480730    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 14:11:15.480792 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:15.480743    2532 projected.go:196] Error preparing data for projected volume kube-api-access-gkbvq for pod openshift-network-diagnostics/network-check-target-pqw2f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:15.480962 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:15.480805    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq podName:be00a0fe-fb34-4722-be6e-6ac18ccaed54 nodeName:}" failed. No retries permitted until 2026-09-09 14:11:23.480783965 +0000 UTC m=+17.243579199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkbvq" (UniqueName: "kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq") pod "network-check-target-pqw2f" (UID: "be00a0fe-fb34-4722-be6e-6ac18ccaed54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:15.728605 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:15.728527    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:15.728763 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:15.728653    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:15.728973 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:15.728527    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:15.728973 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:15.728931    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:16.729064 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:16.729033    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:16.729449 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:16.729169    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:17.728266 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:17.728236    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:17.728454 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:17.728361    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:17.728454 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:17.728400    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:17.728563 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:17.728471    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:18.728811 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:18.728779    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:18.729400 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:18.728877    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:19.728503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:19.728470    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:19.728666 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:19.728476    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:19.728666 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:19.728584    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:19.728790 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:19.728680    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:20.728567 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:20.728534    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:20.728985 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:20.728753    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:21.727975 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:21.727941    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:21.728162 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:21.727941    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:21.728162 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:21.728083    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:21.728162 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:21.728134    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:22.731295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:22.731260    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:22.731744 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:22.731417    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:23.438652 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:23.438621    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:23.438827 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:23.438664    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:23.438827 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:23.438771    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:23.438827 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:23.438823    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs podName:9a8d7ec4-9e2a-40c5-b008-3c6890b621fe nodeName:}" failed. No retries permitted until 2026-09-09 14:11:39.438809497 +0000 UTC m=+33.201604716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs") pod "network-metrics-daemon-lq49r" (UID: "9a8d7ec4-9e2a-40c5-b008-3c6890b621fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:23.438980 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:23.438772    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:23.438980 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:23.438909    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret podName:04d39c37-e950-4def-86cf-bc268ada8e3f nodeName:}" failed. No retries permitted until 2026-09-09 14:11:39.438888319 +0000 UTC m=+33.201683552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret") pod "global-pull-secret-syncer-8xz8x" (UID: "04d39c37-e950-4def-86cf-bc268ada8e3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:23.539496 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:23.539463    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbvq\" (UniqueName: \"kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq\") pod \"network-check-target-pqw2f\" (UID: \"be00a0fe-fb34-4722-be6e-6ac18ccaed54\") " pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:23.539658 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:23.539600    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 14:11:23.539658 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:23.539621    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 14:11:23.539658 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:23.539632    2532 projected.go:196] Error preparing data for projected volume kube-api-access-gkbvq for pod openshift-network-diagnostics/network-check-target-pqw2f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:23.539823 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:23.539694    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq podName:be00a0fe-fb34-4722-be6e-6ac18ccaed54 nodeName:}" failed. No retries permitted until 2026-09-09 14:11:39.539676715 +0000 UTC m=+33.302471933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkbvq" (UniqueName: "kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq") pod "network-check-target-pqw2f" (UID: "be00a0fe-fb34-4722-be6e-6ac18ccaed54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:23.727856 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:23.727766    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:23.727856 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:23.727795    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:23.728069 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:23.727895    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:23.728069 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:23.728038    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:24.731003 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:24.730979    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:24.731410 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:24.731110    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:25.728170 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:25.728143    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:25.728346 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:25.728143    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:25.728346 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:25.728256    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:25.728346 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:25.728329    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:26.728759 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:26.728728    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:26.729241 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:26.728868    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:26.833896 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:26.833663    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p5tl6" event={"ID":"d66b4e8e-2f59-49c1-a992-c3e45fc004a7","Type":"ContainerStarted","Data":"68ecef8b1f15a6c0d9ba79a09c6c12582a540fbd99f7b7377e9d3616bab4d1d3"}
Sep 09 14:11:26.840661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:26.840634    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal" event={"ID":"d6aac1761331084f3b830a859c99fcd9","Type":"ContainerStarted","Data":"aec3f9a70f4eda0e0d739550df654063190cf4e469a999c4de21dbd98aafe421"}
Sep 09 14:11:26.851268 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:26.851160    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p5tl6" podStartSLOduration=1.269072849 podStartE2EDuration="19.851075047s" totalImagesPullingTime="18.582002198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:26.850829506 +0000 UTC m=+20.613624746" watchObservedRunningTime="2026-09-09 14:11:26.851075047 +0000 UTC m=+20.613870284"
Sep 09 14:11:27.728416 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.728210    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:27.728578 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.728255    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:27.728578 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:27.728516    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:27.728689 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:27.728607    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:27.844976 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.844724    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4z47c" event={"ID":"5044a9fc-f558-40d3-9525-f07cb8171534","Type":"ContainerStarted","Data":"eed4b9c7ce34463a1b8cabf9b3ddb6a48e12b8eb2a207376f39e422e3f3623bc"}
Sep 09 14:11:27.846525 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.846497    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-svpv6" event={"ID":"49ea2ecc-a574-4969-85d4-1c6419ce4a06","Type":"ContainerStarted","Data":"4fa3b32d4af0ed41c81d2c9a2aa0ffc376069040a3ab699ac014739c56f83449"}
Sep 09 14:11:27.848078 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.848054    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cn577" event={"ID":"be545f7f-c99c-47c4-9e0e-1e690afab968","Type":"ContainerStarted","Data":"801bbfe472b10e1c5d72db85e06b16b9b096f506504c7885b1925fd47e521b04"}
Sep 09 14:11:27.849630 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.849604    2532 generic.go:356] "Generic (PLEG): container finished" podID="64100743-6529-42aa-8204-8138f81c29c7" containerID="0b431de23effe229d06fa4ca42780fb006efd978e31c78aa9041d65800c4f41d" exitCode=0
Sep 09 14:11:27.849722 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.849677    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5cc9" event={"ID":"64100743-6529-42aa-8204-8138f81c29c7","Type":"ContainerDied","Data":"0b431de23effe229d06fa4ca42780fb006efd978e31c78aa9041d65800c4f41d"}
Sep 09 14:11:27.851561 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.851534    2532 generic.go:356] "Generic (PLEG): container finished" podID="4d1e80e0bdfbb3cd0f5d4bdeeaa8f968" containerID="54ec08b7b77574fdd9b752c9ba56eb67cf93b95a89f4e12ab94a0d916e6d448e" exitCode=0
Sep 09 14:11:27.851652 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.851622    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal" event={"ID":"4d1e80e0bdfbb3cd0f5d4bdeeaa8f968","Type":"ContainerDied","Data":"54ec08b7b77574fdd9b752c9ba56eb67cf93b95a89f4e12ab94a0d916e6d448e"}
Sep 09 14:11:27.853677 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.853654    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dghxg" event={"ID":"bd746049-41ca-4402-93a6-320c0828fcbb","Type":"ContainerStarted","Data":"1e1cae8d54b64203f417258250f4c4401e55fec8734dfcc98f8d1cf7fa00f0ef"}
Sep 09 14:11:27.856901 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.856882    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:11:27.857291 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.857265    2532 generic.go:356] "Generic (PLEG): container finished" podID="8fae84b9-0f9a-4162-84b2-f3488a0e04af" containerID="6ef6648ee889c276a33898ea25d016926b73e900a190ec2d7e5f71139c354933" exitCode=1
Sep 09 14:11:27.857390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.857299    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" event={"ID":"8fae84b9-0f9a-4162-84b2-f3488a0e04af","Type":"ContainerStarted","Data":"18cbcbc93051a5277bea3cf4d18637f6ed6bd81c0b38f4d4df1c170947828bdb"}
Sep 09 14:11:27.857390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.857372    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" event={"ID":"8fae84b9-0f9a-4162-84b2-f3488a0e04af","Type":"ContainerStarted","Data":"406351359a8b71a397473f1ae5e01c27989f42ae2d058a55914d60ec6dc439c9"}
Sep 09 14:11:27.857488 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.857424    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" event={"ID":"8fae84b9-0f9a-4162-84b2-f3488a0e04af","Type":"ContainerStarted","Data":"6c887b69066f3d1c9799dcb91a8503a0017246b3432a157b66e6ec3a5afbd620"}
Sep 09 14:11:27.857488 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.857479    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" event={"ID":"8fae84b9-0f9a-4162-84b2-f3488a0e04af","Type":"ContainerStarted","Data":"22350293af3a04994b1715f0bfc470e70608e697d9ad5750369745a3225e9f67"}
Sep 09 14:11:27.857579 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.857520    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" event={"ID":"8fae84b9-0f9a-4162-84b2-f3488a0e04af","Type":"ContainerDied","Data":"6ef6648ee889c276a33898ea25d016926b73e900a190ec2d7e5f71139c354933"}
Sep 09 14:11:27.857579 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.857570    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" event={"ID":"8fae84b9-0f9a-4162-84b2-f3488a0e04af","Type":"ContainerStarted","Data":"5639b3721aabf2ccb8da0e307c30a6602e6983854df1f5f995373bbc38e65575"}
Sep 09 14:11:27.861525 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.861485    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4z47c" podStartSLOduration=2.287491335 podStartE2EDuration="20.861473415s" totalImagesPullingTime="18.57398208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:27.861255883 +0000 UTC m=+21.624051124" watchObservedRunningTime="2026-09-09 14:11:27.861473415 +0000 UTC m=+21.624268655"
Sep 09 14:11:27.861812 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.861778    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-251.ec2.internal" podStartSLOduration=19.861772802 podStartE2EDuration="19.861772802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:26.86357688 +0000 UTC m=+20.626372122" watchObservedRunningTime="2026-09-09 14:11:27.861772802 +0000 UTC m=+21.624568041"
Sep 09 14:11:27.898847 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.898774    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cn577" podStartSLOduration=2.263927797 podStartE2EDuration="20.898756861s" totalImagesPullingTime="18.634829064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:27.897959136 +0000 UTC m=+21.660754377" watchObservedRunningTime="2026-09-09 14:11:27.898756861 +0000 UTC m=+21.661552104"
Sep 09 14:11:27.928756 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.928720    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-svpv6" podStartSLOduration=2.372938793 podStartE2EDuration="20.9287036s" totalImagesPullingTime="18.555764807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:27.911997465 +0000 UTC m=+21.674792704" watchObservedRunningTime="2026-09-09 14:11:27.9287036 +0000 UTC m=+21.691498844"
Sep 09 14:11:27.944013 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:27.943970    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dghxg" podStartSLOduration=2.363013681 podStartE2EDuration="20.943955621s" totalImagesPullingTime="18.58094194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:27.928595283 +0000 UTC m=+21.691390558" watchObservedRunningTime="2026-09-09 14:11:27.943955621 +0000 UTC m=+21.706750861"
Sep 09 14:11:28.731718 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:28.731519    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:28.731948 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:28.731824    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:28.860732 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:28.860697    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bgmdl" event={"ID":"182b56a9-69f5-455e-a828-37ae40d559d4","Type":"ContainerStarted","Data":"30800a52a9c80582fba8a05d84b75d58d5511518ed1508d014d2d290cbde5882"}
Sep 09 14:11:28.862913 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:28.862881    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal" event={"ID":"4d1e80e0bdfbb3cd0f5d4bdeeaa8f968","Type":"ContainerStarted","Data":"4d65dc3c1dcacb240f667d07f51e1f4ad2be15ddab858a2425fe28b76c1b3583"}
Sep 09 14:11:28.864539 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:28.864503    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw" event={"ID":"154c326a-a75e-49ea-a5f2-6fa03c5cbe55","Type":"ContainerStarted","Data":"ef21508cfb2cf73cce0d5422f5bba24ea6c07a8cbc7ac000eb517991c6a703cb"}
Sep 09 14:11:28.882719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:28.882682    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bgmdl" podStartSLOduration=3.309403443 podStartE2EDuration="21.882670681s" totalImagesPullingTime="18.573267238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:28.882362664 +0000 UTC m=+22.645157894" watchObservedRunningTime="2026-09-09 14:11:28.882670681 +0000 UTC m=+22.645465923"
Sep 09 14:11:28.897618 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:28.897575    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-251.ec2.internal" podStartSLOduration=20.897560551 podStartE2EDuration="20.897560551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:28.897120895 +0000 UTC m=+22.659916135" watchObservedRunningTime="2026-09-09 14:11:28.897560551 +0000 UTC m=+22.660355792"
Sep 09 14:11:29.583995 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:29.583972    2532 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 14:11:29.657937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:29.657851    2532 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T14:11:29.583991324Z","UUID":"baa98237-025d-4b14-8c13-69c1b8b47119","Handler":null,"Name":"","Endpoint":""}
Sep 09 14:11:29.659424 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:29.659403    2532 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 14:11:29.659424 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:29.659439    2532 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 14:11:29.728638 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:29.728580    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:29.728638 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:29.728601    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:29.728769 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:29.728676    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:29.728807 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:29.728784    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:29.868666 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:29.868640    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw" event={"ID":"154c326a-a75e-49ea-a5f2-6fa03c5cbe55","Type":"ContainerStarted","Data":"5755dc8542e1306b2555d725dc464fce0bf72607930e930cd5cdaf0ad239ad4a"}
Sep 09 14:11:29.871433 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:29.871414    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:11:29.871774 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:29.871754    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" event={"ID":"8fae84b9-0f9a-4162-84b2-f3488a0e04af","Type":"ContainerStarted","Data":"11fd569bda2476db243dad39922f87cf7990748f93ca839b5eb8c6910554706f"}
Sep 09 14:11:30.728444 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:30.728414    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:30.728675 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:30.728554    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:30.878741 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:30.878712    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw" event={"ID":"154c326a-a75e-49ea-a5f2-6fa03c5cbe55","Type":"ContainerStarted","Data":"ed6b26d270e93371d4e93fc224137cd40c38dd24ef5f4fa8c9d9deaa7d0e41cd"}
Sep 09 14:11:30.898612 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:30.898566    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm7xw" podStartSLOduration=1.226561545 podStartE2EDuration="23.898553915s" totalImagesPullingTime="22.67199237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=3 observedRunningTime="2026-09-09 14:11:30.898172441 +0000 UTC m=+24.660967681" watchObservedRunningTime="2026-09-09 14:11:30.898553915 +0000 UTC m=+24.661349154"
Sep 09 14:11:31.728167 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:31.728137    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:31.728333 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:31.728172    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:31.728333 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:31.728265    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:31.728450 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:31.728406    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:32.594426 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:32.594397    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:32.594973 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:32.594958    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:32.728786 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:32.728761    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:32.728917 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:32.728897    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:32.886680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:32.886561    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:11:32.887003 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:32.886978    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" event={"ID":"8fae84b9-0f9a-4162-84b2-f3488a0e04af","Type":"ContainerStarted","Data":"6dba3e4207acd1ad33e9230ffd4e639869c8fc15e01680a934a19e2b87f5a472"}
Sep 09 14:11:32.887221 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:32.887195    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:32.887459 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:32.887442    2532 scope.go:117] "RemoveContainer" containerID="6ef6648ee889c276a33898ea25d016926b73e900a190ec2d7e5f71139c354933"
Sep 09 14:11:32.887806 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:32.887780    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-svpv6"
Sep 09 14:11:33.287727 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.287701    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:33.728799 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.728776    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:33.729252 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.728805    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:33.729252 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:33.728869    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:33.729252 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:33.728977    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:33.890680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.890653    2532 generic.go:356] "Generic (PLEG): container finished" podID="64100743-6529-42aa-8204-8138f81c29c7" containerID="c4e15befd11beefdd7384ed52312dfa4c205528451178be03ede93b19aa0b056" exitCode=0
Sep 09 14:11:33.890805 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.890735    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5cc9" event={"ID":"64100743-6529-42aa-8204-8138f81c29c7","Type":"ContainerDied","Data":"c4e15befd11beefdd7384ed52312dfa4c205528451178be03ede93b19aa0b056"}
Sep 09 14:11:33.896531 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.896514    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:11:33.897041 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.897022    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" event={"ID":"8fae84b9-0f9a-4162-84b2-f3488a0e04af","Type":"ContainerStarted","Data":"555892d619728c64984a19aea8b376d20e745a1b727a06392c911eb72f63f3a8"}
Sep 09 14:11:33.897272 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.897130    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:33.897272 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.897200    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:33.912151 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.912110    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:33.912243 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.912212    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:11:33.937842 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:33.937778    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs" podStartSLOduration=8.118610689 podStartE2EDuration="26.937765125s" totalImagesPullingTime="18.819154436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:33.937123484 +0000 UTC m=+27.699918724" watchObservedRunningTime="2026-09-09 14:11:33.937765125 +0000 UTC m=+27.700560365"
Sep 09 14:11:34.728464 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:34.728438    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:34.728589 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:34.728568    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:34.900788 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:34.900729    2532 generic.go:356] "Generic (PLEG): container finished" podID="64100743-6529-42aa-8204-8138f81c29c7" containerID="6e8fd90090c6dc00b6237eb7f1a98c715f7d8ec0dbdcfaa225e9e4ddde548ebb" exitCode=0
Sep 09 14:11:34.901245 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:34.900806    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5cc9" event={"ID":"64100743-6529-42aa-8204-8138f81c29c7","Type":"ContainerDied","Data":"6e8fd90090c6dc00b6237eb7f1a98c715f7d8ec0dbdcfaa225e9e4ddde548ebb"}
Sep 09 14:11:35.728804 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:35.728778    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:35.728895 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:35.728869    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:35.728946 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:35.728923    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:35.729025 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:35.729009    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:35.905384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:35.905326    2532 generic.go:356] "Generic (PLEG): container finished" podID="64100743-6529-42aa-8204-8138f81c29c7" containerID="8a13321aabdcecec1d89e1cd90d543ccc1daa0978751fd9a3fa7eb3e647f0332" exitCode=0
Sep 09 14:11:35.905384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:35.905367    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5cc9" event={"ID":"64100743-6529-42aa-8204-8138f81c29c7","Type":"ContainerDied","Data":"8a13321aabdcecec1d89e1cd90d543ccc1daa0978751fd9a3fa7eb3e647f0332"}
Sep 09 14:11:36.729265 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:36.729235    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:36.729437 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:36.729354    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:37.728057 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:37.728028    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:37.728531 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:37.728028    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:37.728531 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:37.728168    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:37.728531 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:37.728214    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:38.728300 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:38.728273    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:38.728744 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:38.728394    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:39.460170 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:39.460140    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:39.460351 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:39.460188    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:39.460351 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:39.460289    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:39.460351 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:39.460310    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:39.460351 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:39.460351    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret podName:04d39c37-e950-4def-86cf-bc268ada8e3f nodeName:}" failed. No retries permitted until 2026-09-09 14:12:11.460336857 +0000 UTC m=+65.223132074 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret") pod "global-pull-secret-syncer-8xz8x" (UID: "04d39c37-e950-4def-86cf-bc268ada8e3f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 14:11:39.460572 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:39.460367    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs podName:9a8d7ec4-9e2a-40c5-b008-3c6890b621fe nodeName:}" failed. No retries permitted until 2026-09-09 14:12:11.460359941 +0000 UTC m=+65.223155159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs") pod "network-metrics-daemon-lq49r" (UID: "9a8d7ec4-9e2a-40c5-b008-3c6890b621fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 14:11:39.561056 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:39.561021    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbvq\" (UniqueName: \"kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq\") pod \"network-check-target-pqw2f\" (UID: \"be00a0fe-fb34-4722-be6e-6ac18ccaed54\") " pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:39.561214 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:39.561162    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 14:11:39.561214 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:39.561186    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 14:11:39.561214 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:39.561197    2532 projected.go:196] Error preparing data for projected volume kube-api-access-gkbvq for pod openshift-network-diagnostics/network-check-target-pqw2f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:39.561338 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:39.561244    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq podName:be00a0fe-fb34-4722-be6e-6ac18ccaed54 nodeName:}" failed. No retries permitted until 2026-09-09 14:12:11.561232301 +0000 UTC m=+65.324027520 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkbvq" (UniqueName: "kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq") pod "network-check-target-pqw2f" (UID: "be00a0fe-fb34-4722-be6e-6ac18ccaed54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 14:11:39.728245 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:39.728177    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:39.728401 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:39.728286    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:39.728401 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:39.728338    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:39.728772 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:39.728433    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:40.728511 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:40.728482    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:40.729065 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:40.728611    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:41.728207 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:41.728153    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:41.728321 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:41.728153    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:41.728321 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:41.728255    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:41.728321 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:41.728313    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:41.922160 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:41.922133    2532 generic.go:356] "Generic (PLEG): container finished" podID="64100743-6529-42aa-8204-8138f81c29c7" containerID="1ce02f4a50297e57065a555b9f79363394adb9cfa6cfc177d5387e74a91c161c" exitCode=0
Sep 09 14:11:41.922564 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:41.922214    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5cc9" event={"ID":"64100743-6529-42aa-8204-8138f81c29c7","Type":"ContainerDied","Data":"1ce02f4a50297e57065a555b9f79363394adb9cfa6cfc177d5387e74a91c161c"}
Sep 09 14:11:42.728422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:42.728395    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:42.728562 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:42.728492    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:42.927393 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:42.927361    2532 generic.go:356] "Generic (PLEG): container finished" podID="64100743-6529-42aa-8204-8138f81c29c7" containerID="3c7348aa5fce2314690e7e74fd7c25730d234f95c85a86b187b9ac0f1f3d166e" exitCode=0
Sep 09 14:11:42.927732 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:42.927407    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5cc9" event={"ID":"64100743-6529-42aa-8204-8138f81c29c7","Type":"ContainerDied","Data":"3c7348aa5fce2314690e7e74fd7c25730d234f95c85a86b187b9ac0f1f3d166e"}
Sep 09 14:11:43.728498 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:43.728337    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:43.728643 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:43.728335    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:43.728643 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:43.728558    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:43.728643 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:43.728622    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:43.932266 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:43.932235    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5cc9" event={"ID":"64100743-6529-42aa-8204-8138f81c29c7","Type":"ContainerStarted","Data":"f7bcfff81491910d08fa948cd5637cb6adf2f395a900f17855e1190ad55a2cf8"}
Sep 09 14:11:43.957185 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:43.957146    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t5cc9" podStartSLOduration=6.377736801 podStartE2EDuration="36.957131608s" totalImagesPullingTime="30.579394807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:43.956647035 +0000 UTC m=+37.719442275" watchObservedRunningTime="2026-09-09 14:11:43.957131608 +0000 UTC m=+37.719926849"
Sep 09 14:11:44.728557 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:44.728529    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:44.728704 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:44.728624    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:45.728272 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:45.728241    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:45.728690 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:45.728334    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:45.728690 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:45.728389    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:45.728690 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:45.728497    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:46.728699 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:46.728670    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:46.729126 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:46.728769    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:47.728042 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:47.728017    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:47.728208 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:47.728017    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:47.728208 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:47.728119    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:47.728208 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:47.728181    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:48.728467 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:48.728436    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:48.728869 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:48.728560    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:49.728207 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:49.728183    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:49.728331 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:49.728262    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:49.728331 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:49.728275    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:49.728493 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:49.728369    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:49.769287 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:49.769264    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8xz8x"]
Sep 09 14:11:49.772549 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:49.772526    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pqw2f"]
Sep 09 14:11:49.773362 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:49.773340    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lq49r"]
Sep 09 14:11:49.773468 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:49.773456    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:49.773605 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:49.773579    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:49.943586 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:49.943525    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:49.943706 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:49.943525    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:49.943706 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:49.943623    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:49.943706 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:49.943686    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:51.728273 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:51.728240    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:51.728273 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:51.728275    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:51.728683 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:51.728350    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8xz8x" podUID="04d39c37-e950-4def-86cf-bc268ada8e3f"
Sep 09 14:11:51.728683 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:51.728379    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pqw2f" podUID="be00a0fe-fb34-4722-be6e-6ac18ccaed54"
Sep 09 14:11:51.728683 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:51.728413    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:51.728683 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:11:51.728495    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq49r" podUID="9a8d7ec4-9e2a-40c5-b008-3c6890b621fe"
Sep 09 14:11:53.500872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.500853    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-251.ec2.internal" event="NodeReady"
Sep 09 14:11:53.501272 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.500944    2532 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 14:11:53.553588 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.553565    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8b6ld"]
Sep 09 14:11:53.574030 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.574005    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vxgj6"]
Sep 09 14:11:53.574158 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.574137    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.576446 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.576427    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 14:11:53.576549 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.576494    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 14:11:53.576621 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.576584    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tfl8h\""
Sep 09 14:11:53.600108 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.599978    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8b6ld"]
Sep 09 14:11:53.600182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.600136    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vxgj6"]
Sep 09 14:11:53.600182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.600092    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vxgj6"
Sep 09 14:11:53.602421 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.602404    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 14:11:53.602494 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.602424    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 14:11:53.602618 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.602502    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hbr5s\""
Sep 09 14:11:53.602618 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.602523    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 14:11:53.658125 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.658086    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b4wch"]
Sep 09 14:11:53.667509 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.667471    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rldqs\" (UniqueName: \"kubernetes.io/projected/82696d15-9d8e-4912-9084-255e8fac69a4-kube-api-access-rldqs\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.667583 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.667507    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82696d15-9d8e-4912-9084-255e8fac69a4-tmp-dir\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.667583 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.667536    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82696d15-9d8e-4912-9084-255e8fac69a4-config-volume\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.667667 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.667590    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82696d15-9d8e-4912-9084-255e8fac69a4-metrics-tls\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.678093 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.678073    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b4wch"]
Sep 09 14:11:53.678189 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.678178    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.680564 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.680543    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 14:11:53.680790 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.680774    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 14:11:53.680981 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.680967    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 14:11:53.681049 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.681021    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rghsj\""
Sep 09 14:11:53.681120 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.681081    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 14:11:53.728653 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.728631    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:11:53.728653 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.728644    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:11:53.728789 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.728645    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:11:53.731441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.731424    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 14:11:53.731546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.731519    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 14:11:53.731641 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.731624    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 14:11:53.731726 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.731710    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-72t49\""
Sep 09 14:11:53.731785 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.731749    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 14:11:53.731848 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.731836    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f5zw2\""
Sep 09 14:11:53.768444 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768424    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82696d15-9d8e-4912-9084-255e8fac69a4-config-volume\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.768508 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768447    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f63aa86-01bb-4492-b0bd-ed04bf50908f-cert\") pod \"ingress-canary-vxgj6\" (UID: \"0f63aa86-01bb-4492-b0bd-ed04bf50908f\") " pod="openshift-ingress-canary/ingress-canary-vxgj6"
Sep 09 14:11:53.768508 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768473    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82696d15-9d8e-4912-9084-255e8fac69a4-metrics-tls\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.768508 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768492    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-data-volume\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.768636 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768601    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-crio-socket\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.768636 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768620    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.768718 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768645    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.768718 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768671    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rldqs\" (UniqueName: \"kubernetes.io/projected/82696d15-9d8e-4912-9084-255e8fac69a4-kube-api-access-rldqs\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.768838 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768716    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2bhj\" (UniqueName: \"kubernetes.io/projected/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-kube-api-access-x2bhj\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.768920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768899    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcr8s\" (UniqueName: \"kubernetes.io/projected/0f63aa86-01bb-4492-b0bd-ed04bf50908f-kube-api-access-rcr8s\") pod \"ingress-canary-vxgj6\" (UID: \"0f63aa86-01bb-4492-b0bd-ed04bf50908f\") " pod="openshift-ingress-canary/ingress-canary-vxgj6"
Sep 09 14:11:53.768967 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768942    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82696d15-9d8e-4912-9084-255e8fac69a4-tmp-dir\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.769016 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.768980    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82696d15-9d8e-4912-9084-255e8fac69a4-config-volume\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.769173 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.769160    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82696d15-9d8e-4912-9084-255e8fac69a4-tmp-dir\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.772283 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.772261    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82696d15-9d8e-4912-9084-255e8fac69a4-metrics-tls\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.776789 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.776767    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rldqs\" (UniqueName: \"kubernetes.io/projected/82696d15-9d8e-4912-9084-255e8fac69a4-kube-api-access-rldqs\") pod \"dns-default-8b6ld\" (UID: \"82696d15-9d8e-4912-9084-255e8fac69a4\") " pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.869833 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.869813    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-data-volume\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.869915 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.869848    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-crio-socket\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.869915 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.869865    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.869915 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.869892    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.870065 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.869986    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2bhj\" (UniqueName: \"kubernetes.io/projected/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-kube-api-access-x2bhj\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.870065 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.870028    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcr8s\" (UniqueName: \"kubernetes.io/projected/0f63aa86-01bb-4492-b0bd-ed04bf50908f-kube-api-access-rcr8s\") pod \"ingress-canary-vxgj6\" (UID: \"0f63aa86-01bb-4492-b0bd-ed04bf50908f\") " pod="openshift-ingress-canary/ingress-canary-vxgj6"
Sep 09 14:11:53.870195 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.870061    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-crio-socket\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.870195 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.870079    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f63aa86-01bb-4492-b0bd-ed04bf50908f-cert\") pod \"ingress-canary-vxgj6\" (UID: \"0f63aa86-01bb-4492-b0bd-ed04bf50908f\") " pod="openshift-ingress-canary/ingress-canary-vxgj6"
Sep 09 14:11:53.870295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.870214    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-data-volume\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.870490 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.870461    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.871897 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.871869    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.872144 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.872129    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f63aa86-01bb-4492-b0bd-ed04bf50908f-cert\") pod \"ingress-canary-vxgj6\" (UID: \"0f63aa86-01bb-4492-b0bd-ed04bf50908f\") " pod="openshift-ingress-canary/ingress-canary-vxgj6"
Sep 09 14:11:53.878308 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.878287    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2bhj\" (UniqueName: \"kubernetes.io/projected/d9e2b2bb-425b-45a7-9213-67cc4f1e626e-kube-api-access-x2bhj\") pod \"insights-runtime-extractor-b4wch\" (UID: \"d9e2b2bb-425b-45a7-9213-67cc4f1e626e\") " pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:53.878378 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.878309    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcr8s\" (UniqueName: \"kubernetes.io/projected/0f63aa86-01bb-4492-b0bd-ed04bf50908f-kube-api-access-rcr8s\") pod \"ingress-canary-vxgj6\" (UID: \"0f63aa86-01bb-4492-b0bd-ed04bf50908f\") " pod="openshift-ingress-canary/ingress-canary-vxgj6"
Sep 09 14:11:53.883249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.883232    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:53.907908 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.907886    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vxgj6"
Sep 09 14:11:53.986413 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:53.986387    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4wch"
Sep 09 14:11:54.203224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.203163    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8b6ld"]
Sep 09 14:11:54.206174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.206002    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vxgj6"]
Sep 09 14:11:54.207033 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:54.206933    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82696d15_9d8e_4912_9084_255e8fac69a4.slice/crio-acb64728a2b50ec6a236dd050fdb63e2591177aae7b41e8718841c223682a09a WatchSource:0}: Error finding container acb64728a2b50ec6a236dd050fdb63e2591177aae7b41e8718841c223682a09a: Status 404 returned error can't find the container with id acb64728a2b50ec6a236dd050fdb63e2591177aae7b41e8718841c223682a09a
Sep 09 14:11:54.209313 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:54.209289    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f63aa86_01bb_4492_b0bd_ed04bf50908f.slice/crio-d2304473fb1814715772c38dbe49740a9815dd03727faf201c0ee3f0250a5f1e WatchSource:0}: Error finding container d2304473fb1814715772c38dbe49740a9815dd03727faf201c0ee3f0250a5f1e: Status 404 returned error can't find the container with id d2304473fb1814715772c38dbe49740a9815dd03727faf201c0ee3f0250a5f1e
Sep 09 14:11:54.269594 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.269563    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b4wch"]
Sep 09 14:11:54.273872 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:54.273850    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e2b2bb_425b_45a7_9213_67cc4f1e626e.slice/crio-6cdd303bcc03322801c4c0d0f49bf25562638c6ee0a7aba9047b555e0227e06c WatchSource:0}: Error finding container 6cdd303bcc03322801c4c0d0f49bf25562638c6ee0a7aba9047b555e0227e06c: Status 404 returned error can't find the container with id 6cdd303bcc03322801c4c0d0f49bf25562638c6ee0a7aba9047b555e0227e06c
Sep 09 14:11:54.508266 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.508243    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"]
Sep 09 14:11:54.537174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.537150    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-645b7f5d68-vcx9f"]
Sep 09 14:11:54.537297 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.537265    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.539796 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.539766    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 14:11:54.539922 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.539782    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 14:11:54.540038 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.540021    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 14:11:54.540128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.540085    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5czc4\""
Sep 09 14:11:54.540862 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.540846    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 14:11:54.540934 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.540885    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 14:11:54.556912 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.556892    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"]
Sep 09 14:11:54.556992 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.556920    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645b7f5d68-vcx9f"]
Sep 09 14:11:54.557028 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.556989    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.559259 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.559243    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 14:11:54.559350 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.559330    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 14:11:54.559562 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.559545    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 14:11:54.559639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.559586    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 14:11:54.559639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.559632    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 14:11:54.559780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.559759    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 14:11:54.559876 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.559863    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 14:11:54.560058 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.560041    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gvdr8\""
Sep 09 14:11:54.675256 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.675232    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-serving-cert\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.675357 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.675259    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-console-config\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.675357 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.675286    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-oauth-serving-cert\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.675357 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.675313    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8455845f-c049-4206-ac60-a4cf28fd903f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.675357 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.675339    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-service-ca\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.675550 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.675363    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8455845f-c049-4206-ac60-a4cf28fd903f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.675550 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.675388    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6hpq\" (UniqueName: \"kubernetes.io/projected/8455845f-c049-4206-ac60-a4cf28fd903f-kube-api-access-r6hpq\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.675550 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.675430    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-oauth-config\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.675550 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.675458    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8455845f-c049-4206-ac60-a4cf28fd903f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.675550 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.675486    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67sx\" (UniqueName: \"kubernetes.io/projected/5b8da9c6-b99a-48ee-a201-92db914e5482-kube-api-access-w67sx\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.776684 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.776625    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-oauth-config\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.776684 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.776659    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8455845f-c049-4206-ac60-a4cf28fd903f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.776684 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.776677    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67sx\" (UniqueName: \"kubernetes.io/projected/5b8da9c6-b99a-48ee-a201-92db914e5482-kube-api-access-w67sx\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.777020 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.777000    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-serving-cert\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.777070 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.777035    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-console-config\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.777070 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.777065    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-oauth-serving-cert\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.777159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.777091    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8455845f-c049-4206-ac60-a4cf28fd903f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.777159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.777138    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-service-ca\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.777255 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.777161    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8455845f-c049-4206-ac60-a4cf28fd903f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.777255 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.777191    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6hpq\" (UniqueName: \"kubernetes.io/projected/8455845f-c049-4206-ac60-a4cf28fd903f-kube-api-access-r6hpq\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.777453 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.777434    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8455845f-c049-4206-ac60-a4cf28fd903f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.777962 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.777943    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-oauth-serving-cert\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.778179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.778155    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-console-config\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.778272 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.778257    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-service-ca\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.780244 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.780014    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-serving-cert\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.780244 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.780231    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8455845f-c049-4206-ac60-a4cf28fd903f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.780397 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.780140    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-oauth-config\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.780397 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.780080    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8455845f-c049-4206-ac60-a4cf28fd903f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.785470 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.785451    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6hpq\" (UniqueName: \"kubernetes.io/projected/8455845f-c049-4206-ac60-a4cf28fd903f-kube-api-access-r6hpq\") pod \"prometheus-operator-6f4946db56-fhwz7\" (UID: \"8455845f-c049-4206-ac60-a4cf28fd903f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.785563 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.785504    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67sx\" (UniqueName: \"kubernetes.io/projected/5b8da9c6-b99a-48ee-a201-92db914e5482-kube-api-access-w67sx\") pod \"console-645b7f5d68-vcx9f\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") " pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.846232 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.846208    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"
Sep 09 14:11:54.865423 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.865377    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:11:54.956732 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.956702    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4wch" event={"ID":"d9e2b2bb-425b-45a7-9213-67cc4f1e626e","Type":"ContainerStarted","Data":"873190ea2e98e3365d855ee41bdd15280c15f35411373a72d503bf74e95d4b20"}
Sep 09 14:11:54.956895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.956769    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4wch" event={"ID":"d9e2b2bb-425b-45a7-9213-67cc4f1e626e","Type":"ContainerStarted","Data":"6cdd303bcc03322801c4c0d0f49bf25562638c6ee0a7aba9047b555e0227e06c"}
Sep 09 14:11:54.958559 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.958530    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vxgj6" event={"ID":"0f63aa86-01bb-4492-b0bd-ed04bf50908f","Type":"ContainerStarted","Data":"d2304473fb1814715772c38dbe49740a9815dd03727faf201c0ee3f0250a5f1e"}
Sep 09 14:11:54.962224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:54.962194    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8b6ld" event={"ID":"82696d15-9d8e-4912-9084-255e8fac69a4","Type":"ContainerStarted","Data":"acb64728a2b50ec6a236dd050fdb63e2591177aae7b41e8718841c223682a09a"}
Sep 09 14:11:55.144395 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:55.144290    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-fhwz7"]
Sep 09 14:11:55.148017 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:55.147987    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8455845f_c049_4206_ac60_a4cf28fd903f.slice/crio-2ca681b6a5754c94403ba793b0d238fd4ce8be8ebdfbac446692511cf81d46c7 WatchSource:0}: Error finding container 2ca681b6a5754c94403ba793b0d238fd4ce8be8ebdfbac446692511cf81d46c7: Status 404 returned error can't find the container with id 2ca681b6a5754c94403ba793b0d238fd4ce8be8ebdfbac446692511cf81d46c7
Sep 09 14:11:55.168494 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:55.168469    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645b7f5d68-vcx9f"]
Sep 09 14:11:55.173444 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:11:55.173418    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8da9c6_b99a_48ee_a201_92db914e5482.slice/crio-8f502916ce2d14bf5c330a1133a416e1099789105cbc9d79b4f1e5018d2ab762 WatchSource:0}: Error finding container 8f502916ce2d14bf5c330a1133a416e1099789105cbc9d79b4f1e5018d2ab762: Status 404 returned error can't find the container with id 8f502916ce2d14bf5c330a1133a416e1099789105cbc9d79b4f1e5018d2ab762
Sep 09 14:11:55.966683 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:55.966621    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645b7f5d68-vcx9f" event={"ID":"5b8da9c6-b99a-48ee-a201-92db914e5482","Type":"ContainerStarted","Data":"8f502916ce2d14bf5c330a1133a416e1099789105cbc9d79b4f1e5018d2ab762"}
Sep 09 14:11:55.968112 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:55.968065    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7" event={"ID":"8455845f-c049-4206-ac60-a4cf28fd903f","Type":"ContainerStarted","Data":"2ca681b6a5754c94403ba793b0d238fd4ce8be8ebdfbac446692511cf81d46c7"}
Sep 09 14:11:58.981409 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:58.981301    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4wch" event={"ID":"d9e2b2bb-425b-45a7-9213-67cc4f1e626e","Type":"ContainerStarted","Data":"b85aae4767802466531d5f4c58d6e3ec0fa02c4fe44c3593264572a64f63f71d"}
Sep 09 14:11:58.984080 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:58.984036    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vxgj6" event={"ID":"0f63aa86-01bb-4492-b0bd-ed04bf50908f","Type":"ContainerStarted","Data":"c5fe4d91e8799eb1ea5f2d180a22667c20a49d81537e6c4326869235c07965df"}
Sep 09 14:11:58.986235 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:58.986212    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645b7f5d68-vcx9f" event={"ID":"5b8da9c6-b99a-48ee-a201-92db914e5482","Type":"ContainerStarted","Data":"b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b"}
Sep 09 14:11:58.990630 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:58.990604    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8b6ld" event={"ID":"82696d15-9d8e-4912-9084-255e8fac69a4","Type":"ContainerStarted","Data":"837e7a8e62e5740b577ee64457a6989f404da42fe711d147da13c9e63024482f"}
Sep 09 14:11:58.990698 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:58.990658    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8b6ld" event={"ID":"82696d15-9d8e-4912-9084-255e8fac69a4","Type":"ContainerStarted","Data":"f6978d42c83fb8424487ab38cfd5866fd5dc14870d2a61b2499ab5316cab43d1"}
Sep 09 14:11:58.990827 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:58.990812    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:11:58.992658 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:58.992635    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7" event={"ID":"8455845f-c049-4206-ac60-a4cf28fd903f","Type":"ContainerStarted","Data":"8f59c586a5f1cce72ea3d0a37a6fef3393dafcc50e5e6e1dddb5c10cd91ea01d"}
Sep 09 14:11:58.992729 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:58.992681    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7" event={"ID":"8455845f-c049-4206-ac60-a4cf28fd903f","Type":"ContainerStarted","Data":"a1965e265de750e8251f64d7b5dae916613cfbe089793b7d5ee3c80b3c839fcc"}
Sep 09 14:11:59.001888 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:59.001693    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vxgj6" podStartSLOduration=1.46549477 podStartE2EDuration="6.001681883s" totalImagesPullingTime="4.536187113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:59.001480644 +0000 UTC m=+52.764275918" watchObservedRunningTime="2026-09-09 14:11:59.001681883 +0000 UTC m=+52.764477122"
Sep 09 14:11:59.019009 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:59.018770    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-fhwz7" podStartSLOduration=1.4165645040000001 podStartE2EDuration="5.018754211s" totalImagesPullingTime="3.602189707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:59.017493882 +0000 UTC m=+52.780289122" watchObservedRunningTime="2026-09-09 14:11:59.018754211 +0000 UTC m=+52.781549446"
Sep 09 14:11:59.034798 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:59.034723    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-645b7f5d68-vcx9f" podStartSLOduration=1.447373844 podStartE2EDuration="5.034708994s" totalImagesPullingTime="3.58733515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:59.03438176 +0000 UTC m=+52.797177002" watchObservedRunningTime="2026-09-09 14:11:59.034708994 +0000 UTC m=+52.797504236"
Sep 09 14:11:59.054492 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:11:59.053670    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8b6ld" podStartSLOduration=1.515469547 podStartE2EDuration="6.053651227s" totalImagesPullingTime="4.53818168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:11:59.053076474 +0000 UTC m=+52.815871714" watchObservedRunningTime="2026-09-09 14:11:59.053651227 +0000 UTC m=+52.816446468"
Sep 09 14:12:00.851268 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.851235    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"]
Sep 09 14:12:00.853593 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.853574    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:00.856074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.856051    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-55zt2\""
Sep 09 14:12:00.856224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.856172    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 14:12:00.856286 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.856269    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 14:12:00.872492 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.872473    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s7ng9"]
Sep 09 14:12:00.874571 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.874551    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"]
Sep 09 14:12:00.874680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.874587    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"]
Sep 09 14:12:00.874680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.874660    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:00.877080 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.877064    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:00.878170 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.878152    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 14:12:00.878249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.878174    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 14:12:00.878249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.878178    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f2ql4\""
Sep 09 14:12:00.878249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.878194    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 14:12:00.882040 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.882022    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 14:12:00.882292 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.882271    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 14:12:00.882387 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.882273    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 14:12:00.882387 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.882315    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m9rfj\""
Sep 09 14:12:00.895821 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.895802    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"]
Sep 09 14:12:00.932037 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932014    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:00.932145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932042    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-textfile\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:00.932145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932059    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:00.932145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932083    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24a655ce-1c8e-45fb-97cb-00bc12e09e20-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:00.932145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932139    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-wtmp\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:00.932282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932164    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bjpc\" (UniqueName: \"kubernetes.io/projected/24a655ce-1c8e-45fb-97cb-00bc12e09e20-kube-api-access-6bjpc\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:00.932282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932179    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38c179f5-4f8d-4491-a4cd-ceafdc25b295-root\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:00.932282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932197    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-accelerators-collector-config\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:00.932282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932235    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-tls\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:00.932282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932261    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a655ce-1c8e-45fb-97cb-00bc12e09e20-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:00.932422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932290    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:00.932422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932311    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:00.932422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932329    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24a655ce-1c8e-45fb-97cb-00bc12e09e20-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:00.932422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932387    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwdr\" (UniqueName: \"kubernetes.io/projected/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-api-access-4nwdr\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:00.932536 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932410    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38c179f5-4f8d-4491-a4cd-ceafdc25b295-sys\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:00.932536 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932442    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5vtv\" (UniqueName: \"kubernetes.io/projected/38c179f5-4f8d-4491-a4cd-ceafdc25b295-kube-api-access-t5vtv\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:00.932536 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932460    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:00.932536 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932473    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38c179f5-4f8d-4491-a4cd-ceafdc25b295-metrics-client-ca\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:00.932536 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:00.932495    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.001661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.001637    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4wch" event={"ID":"d9e2b2bb-425b-45a7-9213-67cc4f1e626e","Type":"ContainerStarted","Data":"b7a0531cea8411dd18867fee9785a5422b955d3ad7391e31c9fa07be57bb021f"}
Sep 09 14:12:01.019783 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.019718    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b4wch" podStartSLOduration=2.370666169 podStartE2EDuration="8.019706983s" totalImagesPullingTime="5.649040814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:01.018832218 +0000 UTC m=+54.781627459" watchObservedRunningTime="2026-09-09 14:12:01.019706983 +0000 UTC m=+54.782502224"
Sep 09 14:12:01.032767 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.032748    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.032848 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.032774    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38c179f5-4f8d-4491-a4cd-ceafdc25b295-metrics-client-ca\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.032848 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.032793    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.032951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.032852    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.032951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.032878    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-textfile\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.032951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.032903    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.032951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.032939    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24a655ce-1c8e-45fb-97cb-00bc12e09e20-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:01.033159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.032966    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-wtmp\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.033159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.032992    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bjpc\" (UniqueName: \"kubernetes.io/projected/24a655ce-1c8e-45fb-97cb-00bc12e09e20-kube-api-access-6bjpc\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:01.033159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033013    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38c179f5-4f8d-4491-a4cd-ceafdc25b295-root\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.033159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033042    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-accelerators-collector-config\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.033159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033068    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-tls\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.033159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033116    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a655ce-1c8e-45fb-97cb-00bc12e09e20-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:01.033449 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033187    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.033449 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033209    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.033449 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033239    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24a655ce-1c8e-45fb-97cb-00bc12e09e20-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:01.033449 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033312    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwdr\" (UniqueName: \"kubernetes.io/projected/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-api-access-4nwdr\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.033449 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033337    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38c179f5-4f8d-4491-a4cd-ceafdc25b295-sys\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.033449 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033357    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5vtv\" (UniqueName: \"kubernetes.io/projected/38c179f5-4f8d-4491-a4cd-ceafdc25b295-kube-api-access-t5vtv\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.033449 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033417    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-textfile\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.033768 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.033533    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-wtmp\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.034834 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.034765    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-accelerators-collector-config\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.034927 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.034871    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.035731 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.035001    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24a655ce-1c8e-45fb-97cb-00bc12e09e20-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:01.035731 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.035636    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38c179f5-4f8d-4491-a4cd-ceafdc25b295-root\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.035731 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.035693    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38c179f5-4f8d-4491-a4cd-ceafdc25b295-sys\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.035731 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:12:01.035709    2532 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 09 14:12:01.039841 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:12:01.035793    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a655ce-1c8e-45fb-97cb-00bc12e09e20-openshift-state-metrics-tls podName:24a655ce-1c8e-45fb-97cb-00bc12e09e20 nodeName:}" failed. No retries permitted until 2026-09-09 14:12:01.535767198 +0000 UTC m=+55.298562421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/24a655ce-1c8e-45fb-97cb-00bc12e09e20-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-tnrxq" (UID: "24a655ce-1c8e-45fb-97cb-00bc12e09e20") : secret "openshift-state-metrics-tls" not found
Sep 09 14:12:01.039841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.036009    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.039841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.036134    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.039841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.035353    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38c179f5-4f8d-4491-a4cd-ceafdc25b295-metrics-client-ca\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.039841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.037600    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-tls\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.039841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.039828    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38c179f5-4f8d-4491-a4cd-ceafdc25b295-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.041045 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.041024    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24a655ce-1c8e-45fb-97cb-00bc12e09e20-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:01.041153 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.041125    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.041454 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.041430    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.043251 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.043233    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5vtv\" (UniqueName: \"kubernetes.io/projected/38c179f5-4f8d-4491-a4cd-ceafdc25b295-kube-api-access-t5vtv\") pod \"node-exporter-s7ng9\" (UID: \"38c179f5-4f8d-4491-a4cd-ceafdc25b295\") " pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.043488 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.043467    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwdr\" (UniqueName: \"kubernetes.io/projected/4ae0ea2c-ab1f-469c-a331-bfc1cab73317-kube-api-access-4nwdr\") pod \"kube-state-metrics-74cb57df7-25xwv\" (UID: \"4ae0ea2c-ab1f-469c-a331-bfc1cab73317\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.043529 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.043467    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bjpc\" (UniqueName: \"kubernetes.io/projected/24a655ce-1c8e-45fb-97cb-00bc12e09e20-kube-api-access-6bjpc\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:01.185567 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.185516    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s7ng9"
Sep 09 14:12:01.190017 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.189997    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"
Sep 09 14:12:01.195928 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:01.195904    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c179f5_4f8d_4491_a4cd_ceafdc25b295.slice/crio-3a2a86ffd7dbee8bb791d463ec777f94f08391e776bfa36d5ac57d3e9d449541 WatchSource:0}: Error finding container 3a2a86ffd7dbee8bb791d463ec777f94f08391e776bfa36d5ac57d3e9d449541: Status 404 returned error can't find the container with id 3a2a86ffd7dbee8bb791d463ec777f94f08391e776bfa36d5ac57d3e9d449541
Sep 09 14:12:01.460007 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.459934    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-25xwv"]
Sep 09 14:12:01.462524 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:01.462498    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae0ea2c_ab1f_469c_a331_bfc1cab73317.slice/crio-70d496c237795dc56fa2d37de4c7f9cf489349473b310c5b152d7545d3d4a2d0 WatchSource:0}: Error finding container 70d496c237795dc56fa2d37de4c7f9cf489349473b310c5b152d7545d3d4a2d0: Status 404 returned error can't find the container with id 70d496c237795dc56fa2d37de4c7f9cf489349473b310c5b152d7545d3d4a2d0
Sep 09 14:12:01.536958 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.536934    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a655ce-1c8e-45fb-97cb-00bc12e09e20-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:01.538990 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.538972    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24a655ce-1c8e-45fb-97cb-00bc12e09e20-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-tnrxq\" (UID: \"24a655ce-1c8e-45fb-97cb-00bc12e09e20\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:01.762786 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.762756    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"
Sep 09 14:12:01.945071 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.945042    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 14:12:01.948628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.948598    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:01.951012 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.950989    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 14:12:01.951228 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.951211    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 14:12:01.951325 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.951258    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 14:12:01.952311 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.952291    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 14:12:01.952410 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.952299    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 14:12:01.952895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.952795    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 14:12:01.952895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.952824    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 14:12:01.953077 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.952953    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 14:12:01.953077 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.952978    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 14:12:01.953595 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.953575    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-w7c5z\""
Sep 09 14:12:01.961646 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:01.961629    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 14:12:02.011118 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.010300    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv" event={"ID":"4ae0ea2c-ab1f-469c-a331-bfc1cab73317","Type":"ContainerStarted","Data":"70d496c237795dc56fa2d37de4c7f9cf489349473b310c5b152d7545d3d4a2d0"}
Sep 09 14:12:02.012859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.012681    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s7ng9" event={"ID":"38c179f5-4f8d-4491-a4cd-ceafdc25b295","Type":"ContainerStarted","Data":"3a2a86ffd7dbee8bb791d463ec777f94f08391e776bfa36d5ac57d3e9d449541"}
Sep 09 14:12:02.042598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042313    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042359    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-volume\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042434    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042471    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042497    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-out\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042523    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042554    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042580    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042605    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042989 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042628    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw6gz\" (UniqueName: \"kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-kube-api-access-tw6gz\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042989 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042660    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042989 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042689    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-web-config\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.042989 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042731    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq"]
Sep 09 14:12:02.042989 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.042779    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.096758 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:02.096727    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a655ce_1c8e_45fb_97cb_00bc12e09e20.slice/crio-9a58f570a36dd4fa5a9a39b7682862c0a849e91526cf4385f91294fa8b7fee76 WatchSource:0}: Error finding container 9a58f570a36dd4fa5a9a39b7682862c0a849e91526cf4385f91294fa8b7fee76: Status 404 returned error can't find the container with id 9a58f570a36dd4fa5a9a39b7682862c0a849e91526cf4385f91294fa8b7fee76
Sep 09 14:12:02.143819 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.143787    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.143962 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.143842    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.143962 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.143877    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-out\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.143962 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.143908    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.143962 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.143939    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.144261 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:12:02.143961    2532 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 14:12:02.144261 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:12:02.144030    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-main-tls podName:1922ab8d-b3e6-406d-b87c-36a73a7387dd nodeName:}" failed. No retries permitted until 2026-09-09 14:12:02.644007861 +0000 UTC m=+56.406803094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd") : secret "alertmanager-main-tls" not found
Sep 09 14:12:02.144261 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.143968    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.144261 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.144077    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.145679 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.144124    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw6gz\" (UniqueName: \"kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-kube-api-access-tw6gz\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.145679 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.144522    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.145679 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:12:02.144550    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-trusted-ca-bundle podName:1922ab8d-b3e6-406d-b87c-36a73a7387dd nodeName:}" failed. No retries permitted until 2026-09-09 14:12:02.644532531 +0000 UTC m=+56.407327768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd") : configmap references non-existent config key: ca-bundle.crt
Sep 09 14:12:02.145679 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.144584    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-web-config\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.145679 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.144663    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.145679 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.144697    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.145679 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.144725    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-volume\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.145679 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.144805    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.145679 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.145416    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.148548 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.148494    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.148653 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.148560    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.148653 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.148617    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-web-config\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.149443 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.149410    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.149846 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.149814    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.149962 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.149940    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-out\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.150166 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.150134    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-volume\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.150411 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.150393    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.153637 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.153615    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw6gz\" (UniqueName: \"kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-kube-api-access-tw6gz\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.649675 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.649592    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.649917 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.649834    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.650638 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.650589    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.652480 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.652461    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:02.860738 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:02.860705    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:12:03.020758 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.020725    2532 generic.go:356] "Generic (PLEG): container finished" podID="38c179f5-4f8d-4491-a4cd-ceafdc25b295" containerID="53cea5c15629ccffab9202a9e39d9a5356da2de0c2e566f5454ec2b4bbc8b02f" exitCode=0
Sep 09 14:12:03.021220 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.020805    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s7ng9" event={"ID":"38c179f5-4f8d-4491-a4cd-ceafdc25b295","Type":"ContainerDied","Data":"53cea5c15629ccffab9202a9e39d9a5356da2de0c2e566f5454ec2b4bbc8b02f"}
Sep 09 14:12:03.022887 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.022865    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq" event={"ID":"24a655ce-1c8e-45fb-97cb-00bc12e09e20","Type":"ContainerStarted","Data":"18642ed104f757bd897a6a2c3eac99d500974a36b0cb3f7b15549752bb7cc5c8"}
Sep 09 14:12:03.022986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.022905    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq" event={"ID":"24a655ce-1c8e-45fb-97cb-00bc12e09e20","Type":"ContainerStarted","Data":"6c85a750d091d217a05482e5057739a7484b72993020c70d5987fa7d82a3ca28"}
Sep 09 14:12:03.022986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.022923    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq" event={"ID":"24a655ce-1c8e-45fb-97cb-00bc12e09e20","Type":"ContainerStarted","Data":"9a58f570a36dd4fa5a9a39b7682862c0a849e91526cf4385f91294fa8b7fee76"}
Sep 09 14:12:03.140426 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.140396    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 14:12:03.144493 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:03.144155    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1922ab8d_b3e6_406d_b87c_36a73a7387dd.slice/crio-f7601769ec1fcf6a3fbd0df70568dbaa715f568b5efe989e8800cf9b2b6ae471 WatchSource:0}: Error finding container f7601769ec1fcf6a3fbd0df70568dbaa715f568b5efe989e8800cf9b2b6ae471: Status 404 returned error can't find the container with id f7601769ec1fcf6a3fbd0df70568dbaa715f568b5efe989e8800cf9b2b6ae471
Sep 09 14:12:03.432330 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.432255    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-556794d8df-488p4"]
Sep 09 14:12:03.448924 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.448891    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556794d8df-488p4"]
Sep 09 14:12:03.449073 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.449009    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.456907 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.456882    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 14:12:03.559115 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.559077    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-oauth-config\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.559295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.559135    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-service-ca\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.559295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.559163    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwzj8\" (UniqueName: \"kubernetes.io/projected/ce04d435-e1d7-494d-8908-685c0f03befa-kube-api-access-nwzj8\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.559295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.559185    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-console-config\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.559295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.559201    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-oauth-serving-cert\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.559295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.559217    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-serving-cert\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.559295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.559231    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-trusted-ca-bundle\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.660197 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.660168    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-oauth-config\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.660197 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.660197    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-service-ca\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.660379 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.660222    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzj8\" (UniqueName: \"kubernetes.io/projected/ce04d435-e1d7-494d-8908-685c0f03befa-kube-api-access-nwzj8\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.660379 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.660243    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-console-config\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.660379 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.660257    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-oauth-serving-cert\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.660379 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.660276    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-serving-cert\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.660379 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.660298    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-trusted-ca-bundle\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.661016 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.660983    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-service-ca\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.661142 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.661063    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-console-config\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.661142 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.661063    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-oauth-serving-cert\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.661229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.661213    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-trusted-ca-bundle\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.663239 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.663210    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-oauth-config\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.663321 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.663219    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-serving-cert\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.671081 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.671057    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzj8\" (UniqueName: \"kubernetes.io/projected/ce04d435-e1d7-494d-8908-685c0f03befa-kube-api-access-nwzj8\") pod \"console-556794d8df-488p4\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") " pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.758342 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.758320    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:03.881492 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.881237    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-58578cccb4-8jclj"]
Sep 09 14:12:03.894961 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.894938    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:03.897935 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.897660    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-58578cccb4-8jclj"]
Sep 09 14:12:03.897935 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.897708    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 14:12:03.897935 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.897803    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 14:12:03.897935 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.897850    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 14:12:03.897935 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.897868    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 14:12:03.897935 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.897884    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p4kft\""
Sep 09 14:12:03.898307 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.898002    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 14:12:03.898307 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.898131    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eqfvq4tg2uhoo\""
Sep 09 14:12:03.963478 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.963440    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wdv\" (UniqueName: \"kubernetes.io/projected/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-kube-api-access-m9wdv\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:03.963608 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.963528    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:03.963608 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.963558    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:03.963715 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.963655    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-metrics-client-ca\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:03.963715 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.963687    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-grpc-tls\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:03.963806 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.963711    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-tls\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:03.963806 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.963742    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:03.963918 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:03.963802    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.029609 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.029539    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv" event={"ID":"4ae0ea2c-ab1f-469c-a331-bfc1cab73317","Type":"ContainerStarted","Data":"eb8cc908a4c04fee383891729b3b619e39c8632839210fcbb61b16760dec3299"}
Sep 09 14:12:04.030023 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.029609    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv" event={"ID":"4ae0ea2c-ab1f-469c-a331-bfc1cab73317","Type":"ContainerStarted","Data":"9a4593f6fd9915528fd8f60409ecbae5546dd917cbcdb4c322a001e9dc8d41f7"}
Sep 09 14:12:04.030023 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.029646    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv" event={"ID":"4ae0ea2c-ab1f-469c-a331-bfc1cab73317","Type":"ContainerStarted","Data":"8729ef7fdd821c9c0220825a636be266d2047d8216b6827749773c53f9e9617b"}
Sep 09 14:12:04.032207 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.032181    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s7ng9" event={"ID":"38c179f5-4f8d-4491-a4cd-ceafdc25b295","Type":"ContainerStarted","Data":"98f876f5a70f024c6a1b446f30407fc1574a806f0b76d2b30fc092f9ddedf48f"}
Sep 09 14:12:04.032326 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.032229    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s7ng9" event={"ID":"38c179f5-4f8d-4491-a4cd-ceafdc25b295","Type":"ContainerStarted","Data":"f27cd3d627efd9fab0a62bee93c121d5c956f0a62eb53f660eb7495bc5395fa7"}
Sep 09 14:12:04.033796 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.033681    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerStarted","Data":"f7601769ec1fcf6a3fbd0df70568dbaa715f568b5efe989e8800cf9b2b6ae471"}
Sep 09 14:12:04.056240 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.056171    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-25xwv" podStartSLOduration=2.497506435 podStartE2EDuration="4.05615604s" totalImagesPullingTime="1.558649605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:04.055113138 +0000 UTC m=+57.817908402" watchObservedRunningTime="2026-09-09 14:12:04.05615604 +0000 UTC m=+57.818951292"
Sep 09 14:12:04.063229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.063199    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556794d8df-488p4"]
Sep 09 14:12:04.064677 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.064657    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-metrics-client-ca\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.064762 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.064683    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-grpc-tls\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.064762 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.064710    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-tls\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.065523 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.065486    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-metrics-client-ca\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.065816 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.064755    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.065816 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.065689    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.067825 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.067799    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wdv\" (UniqueName: \"kubernetes.io/projected/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-kube-api-access-m9wdv\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.068012 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.067993    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.068092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.068033    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.068334 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.068303    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.071054 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.069777    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-grpc-tls\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.071822 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.071283    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.073592 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.073275    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.075614 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.075589    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.077365 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.077320    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-secret-thanos-querier-tls\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.077762 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.077642    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s7ng9" podStartSLOduration=3.074822884 podStartE2EDuration="4.077624774s" totalImagesPullingTime="929.288527ms" totalInitContainerRuntime="73.513363ms" isStatefulPod=true podCreationTimestamp="2026-09-09 14:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:04.07680338 +0000 UTC m=+57.839598621" watchObservedRunningTime="2026-09-09 14:12:04.077624774 +0000 UTC m=+57.840420015"
Sep 09 14:12:04.079766 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.079748    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wdv\" (UniqueName: \"kubernetes.io/projected/bd26eb7e-40ac-4286-9ff7-6a2dead1d728-kube-api-access-m9wdv\") pod \"thanos-querier-58578cccb4-8jclj\" (UID: \"bd26eb7e-40ac-4286-9ff7-6a2dead1d728\") " pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.208812 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.208759    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:04.723251 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.723188    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-58578cccb4-8jclj"]
Sep 09 14:12:04.762086 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:04.762055    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd26eb7e_40ac_4286_9ff7_6a2dead1d728.slice/crio-95728c2336c0fa9b8acab0f45834674ad3b72476e3001b7f5d7417fbab0e684b WatchSource:0}: Error finding container 95728c2336c0fa9b8acab0f45834674ad3b72476e3001b7f5d7417fbab0e684b: Status 404 returned error can't find the container with id 95728c2336c0fa9b8acab0f45834674ad3b72476e3001b7f5d7417fbab0e684b
Sep 09 14:12:04.865901 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.865864    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:12:04.866011 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.865935    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:12:04.871280 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:04.871260    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:12:05.041017 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.040979    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj" event={"ID":"bd26eb7e-40ac-4286-9ff7-6a2dead1d728","Type":"ContainerStarted","Data":"95728c2336c0fa9b8acab0f45834674ad3b72476e3001b7f5d7417fbab0e684b"}
Sep 09 14:12:05.043340 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.043312    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq" event={"ID":"24a655ce-1c8e-45fb-97cb-00bc12e09e20","Type":"ContainerStarted","Data":"f02b2f4a1703a9205b236837ee4f74efddf035b0bc8339538752517394711e02"}
Sep 09 14:12:05.044980 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.044947    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556794d8df-488p4" event={"ID":"ce04d435-e1d7-494d-8908-685c0f03befa","Type":"ContainerStarted","Data":"3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4"}
Sep 09 14:12:05.045108 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.044992    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556794d8df-488p4" event={"ID":"ce04d435-e1d7-494d-8908-685c0f03befa","Type":"ContainerStarted","Data":"c959bc735da1c941f9e8a2edff8177b88fe5eddec9c6db7ac1a2d13510f6d104"}
Sep 09 14:12:05.050205 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.050185    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:12:05.061867 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.061828    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-tnrxq" podStartSLOduration=2.475255121 podStartE2EDuration="5.061812815s" totalImagesPullingTime="2.586557694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:05.060857689 +0000 UTC m=+58.823652929" watchObservedRunningTime="2026-09-09 14:12:05.061812815 +0000 UTC m=+58.824608055"
Sep 09 14:12:05.093753 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.093702    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-556794d8df-488p4" podStartSLOduration=2.093683106 podStartE2EDuration="2.093683106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:05.092140406 +0000 UTC m=+58.854935647" watchObservedRunningTime="2026-09-09 14:12:05.093683106 +0000 UTC m=+58.856478347"
Sep 09 14:12:05.288963 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.288939    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f49bd8f8-zx26s"]
Sep 09 14:12:05.297273 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.297257    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.300237 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.300217    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 14:12:05.300508 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.300364    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5flt7\""
Sep 09 14:12:05.300508 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.300373    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 14:12:05.300508 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.300446    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 14:12:05.300508 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.300475    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 14:12:05.300753 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.300737    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-atcu2gv8p579i\""
Sep 09 14:12:05.303558 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.303540    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f49bd8f8-zx26s"]
Sep 09 14:12:05.380521 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.380498    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-metrics-server-audit-profiles\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.380607 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.380540    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-audit-log\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.380607 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.380569    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.380675 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.380637    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-secret-metrics-server-client-certs\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.380724 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.380711    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-secret-metrics-server-tls\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.380756 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.380746    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-client-ca-bundle\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.380793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.380775    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vd5\" (UniqueName: \"kubernetes.io/projected/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-kube-api-access-69vd5\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.481198 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.481173    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-secret-metrics-server-client-certs\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.481317 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.481239    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-secret-metrics-server-tls\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.481317 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.481264    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-client-ca-bundle\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.481441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.481410    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vd5\" (UniqueName: \"kubernetes.io/projected/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-kube-api-access-69vd5\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.481487 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.481442    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-metrics-server-audit-profiles\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.481628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.481599    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-audit-log\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.481747 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.481663    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.481978 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.481945    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-audit-log\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.482269 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.482252    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.482524 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.482501    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-metrics-server-audit-profiles\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.483642 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.483622    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-client-ca-bundle\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.483707 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.483665    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-secret-metrics-server-tls\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.483740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.483720    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-secret-metrics-server-client-certs\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.489658 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.489641    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vd5\" (UniqueName: \"kubernetes.io/projected/7a9e9fb2-cb35-4732-b75d-4db5e66b3f27-kube-api-access-69vd5\") pod \"metrics-server-7f49bd8f8-zx26s\" (UID: \"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27\") " pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.622965 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.622909    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:05.636943 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.636922    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"]
Sep 09 14:12:05.639224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.639207    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:05.641672 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.641652    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 14:12:05.641812 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.641697    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dpv5t\""
Sep 09 14:12:05.655481 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.655457    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"]
Sep 09 14:12:05.683425 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.683396    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a68a09d4-7a18-44db-bcbc-3e2e3946c5f7-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-jw8nl\" (UID: \"a68a09d4-7a18-44db-bcbc-3e2e3946c5f7\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:05.683538 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.683433    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgssn\" (UniqueName: \"kubernetes.io/projected/a68a09d4-7a18-44db-bcbc-3e2e3946c5f7-kube-api-access-fgssn\") pod \"monitoring-plugin-85bbbbbf4-jw8nl\" (UID: \"a68a09d4-7a18-44db-bcbc-3e2e3946c5f7\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:05.784265 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.784231    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a68a09d4-7a18-44db-bcbc-3e2e3946c5f7-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-jw8nl\" (UID: \"a68a09d4-7a18-44db-bcbc-3e2e3946c5f7\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:05.784265 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.784268    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgssn\" (UniqueName: \"kubernetes.io/projected/a68a09d4-7a18-44db-bcbc-3e2e3946c5f7-kube-api-access-fgssn\") pod \"monitoring-plugin-85bbbbbf4-jw8nl\" (UID: \"a68a09d4-7a18-44db-bcbc-3e2e3946c5f7\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:05.784468 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:12:05.784406    2532 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 14:12:05.784518 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:12:05.784481    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a68a09d4-7a18-44db-bcbc-3e2e3946c5f7-monitoring-plugin-cert podName:a68a09d4-7a18-44db-bcbc-3e2e3946c5f7 nodeName:}" failed. No retries permitted until 2026-09-09 14:12:06.284461137 +0000 UTC m=+60.047256370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a68a09d4-7a18-44db-bcbc-3e2e3946c5f7-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-jw8nl" (UID: "a68a09d4-7a18-44db-bcbc-3e2e3946c5f7") : secret "monitoring-plugin-cert" not found
Sep 09 14:12:05.793538 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.793516    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgssn\" (UniqueName: \"kubernetes.io/projected/a68a09d4-7a18-44db-bcbc-3e2e3946c5f7-kube-api-access-fgssn\") pod \"monitoring-plugin-85bbbbbf4-jw8nl\" (UID: \"a68a09d4-7a18-44db-bcbc-3e2e3946c5f7\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:05.896090 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.896067    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f49bd8f8-zx26s"]
Sep 09 14:12:05.898395 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:05.898361    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9e9fb2_cb35_4732_b75d_4db5e66b3f27.slice/crio-4f12bb3970e3a1fce32bb1ed9973f0a4ee601eae411835eb8e8fd251c54d79ea WatchSource:0}: Error finding container 4f12bb3970e3a1fce32bb1ed9973f0a4ee601eae411835eb8e8fd251c54d79ea: Status 404 returned error can't find the container with id 4f12bb3970e3a1fce32bb1ed9973f0a4ee601eae411835eb8e8fd251c54d79ea
Sep 09 14:12:05.916911 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:05.916887    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxnxs"
Sep 09 14:12:06.051937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.051896    2532 generic.go:356] "Generic (PLEG): container finished" podID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerID="8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f" exitCode=0
Sep 09 14:12:06.052370 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.051996    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerDied","Data":"8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f"}
Sep 09 14:12:06.053527 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.053495    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s" event={"ID":"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27","Type":"ContainerStarted","Data":"4f12bb3970e3a1fce32bb1ed9973f0a4ee601eae411835eb8e8fd251c54d79ea"}
Sep 09 14:12:06.105642 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.105616    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d859d47f9-4pk4c"]
Sep 09 14:12:06.108654 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.108633    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.110917 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.110897    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 14:12:06.111240 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.111217    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 14:12:06.111240 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.111236    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-z4vl4\""
Sep 09 14:12:06.111384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.111364    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 14:12:06.111440 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.111385    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 14:12:06.111487 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.111447    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 14:12:06.118060 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.118040    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 14:12:06.121530 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.121507    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d859d47f9-4pk4c"]
Sep 09 14:12:06.189161 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.189074    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-telemeter-client-tls\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.189161 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.189132    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-federate-client-tls\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.189383 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.189164    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b25de1ba-24ef-4c06-8cba-957b49915805-metrics-client-ca\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.189383 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.189204    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25de1ba-24ef-4c06-8cba-957b49915805-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.189383 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.189280    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.189383 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.189324    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-secret-telemeter-client\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.189564 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.189380    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25de1ba-24ef-4c06-8cba-957b49915805-serving-certs-ca-bundle\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.189564 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.189431    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhn5\" (UniqueName: \"kubernetes.io/projected/b25de1ba-24ef-4c06-8cba-957b49915805-kube-api-access-jfhn5\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.290696 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.290661    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.290879 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.290724    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-secret-telemeter-client\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.290879 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.290765    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25de1ba-24ef-4c06-8cba-957b49915805-serving-certs-ca-bundle\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.290879 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.290795    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhn5\" (UniqueName: \"kubernetes.io/projected/b25de1ba-24ef-4c06-8cba-957b49915805-kube-api-access-jfhn5\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.291042 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.290883    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a68a09d4-7a18-44db-bcbc-3e2e3946c5f7-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-jw8nl\" (UID: \"a68a09d4-7a18-44db-bcbc-3e2e3946c5f7\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:06.291042 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.290911    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-telemeter-client-tls\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.291042 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.290954    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-federate-client-tls\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.291042 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.290983    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b25de1ba-24ef-4c06-8cba-957b49915805-metrics-client-ca\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.291042 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.291007    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25de1ba-24ef-4c06-8cba-957b49915805-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.292821 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.292492    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b25de1ba-24ef-4c06-8cba-957b49915805-metrics-client-ca\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.292821 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.292563    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25de1ba-24ef-4c06-8cba-957b49915805-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.292821 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.292700    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25de1ba-24ef-4c06-8cba-957b49915805-serving-certs-ca-bundle\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.294027 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.293591    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.294027 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.293990    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-federate-client-tls\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.294735 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.294710    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-telemeter-client-tls\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.294833 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.294766    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a68a09d4-7a18-44db-bcbc-3e2e3946c5f7-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-jw8nl\" (UID: \"a68a09d4-7a18-44db-bcbc-3e2e3946c5f7\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:06.295419 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.295399    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b25de1ba-24ef-4c06-8cba-957b49915805-secret-telemeter-client\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.301768 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.301740    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhn5\" (UniqueName: \"kubernetes.io/projected/b25de1ba-24ef-4c06-8cba-957b49915805-kube-api-access-jfhn5\") pod \"telemeter-client-d859d47f9-4pk4c\" (UID: \"b25de1ba-24ef-4c06-8cba-957b49915805\") " pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.420747 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.420713    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c"
Sep 09 14:12:06.554893 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.554865    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:06.698689 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.698639    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d859d47f9-4pk4c"]
Sep 09 14:12:06.832144 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:06.832115    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"]
Sep 09 14:12:07.107964 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.107890    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 14:12:07.115171 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.114145    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.117776 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.117701    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qrsg5\""
Sep 09 14:12:07.118235 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.117707    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 14:12:07.118641 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.118619    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 14:12:07.118882 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.118856    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 14:12:07.119122 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.119088    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5np16ljfo8h4j\""
Sep 09 14:12:07.119393 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.119370    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 14:12:07.119521 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.119200    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 14:12:07.119825 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.119788    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 14:12:07.122589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.121266    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 14:12:07.122589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.121487    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 14:12:07.122589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.121693    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 14:12:07.122589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.121915    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 14:12:07.122589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.122168    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 14:12:07.122589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.122404    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 14:12:07.122956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.122769    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 14:12:07.127892 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.127628    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 14:12:07.128427 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.128406    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 14:12:07.200645 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.200567    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.200645 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.200611    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.200864 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.200719    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.200864 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.200751    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.200864 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.200787    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.200860    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.200902    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-config\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.200930    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.200955    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.200977    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.201019    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.201042    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.201066    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201558 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.201317    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201558 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.201372    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-config-out\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201558 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.201403    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201558 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.201446    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201558 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.201476    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-web-config\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201558 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.201504    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.201558 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.201541    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6527p\" (UniqueName: \"kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-kube-api-access-6527p\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.256361 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:07.256325    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68a09d4_7a18_44db_bcbc_3e2e3946c5f7.slice/crio-fdd9e4e7101a3c860d6dd679be3c3151918b2ca30879ece88237cd5784e6a588 WatchSource:0}: Error finding container fdd9e4e7101a3c860d6dd679be3c3151918b2ca30879ece88237cd5784e6a588: Status 404 returned error can't find the container with id fdd9e4e7101a3c860d6dd679be3c3151918b2ca30879ece88237cd5784e6a588
Sep 09 14:12:07.256792 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:07.256745    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25de1ba_24ef_4c06_8cba_957b49915805.slice/crio-fc62bf252b1b8b69d7b7daf1a4cfba903e5bc597762d51abd8924caa0f5bee79 WatchSource:0}: Error finding container fc62bf252b1b8b69d7b7daf1a4cfba903e5bc597762d51abd8924caa0f5bee79: Status 404 returned error can't find the container with id fc62bf252b1b8b69d7b7daf1a4cfba903e5bc597762d51abd8924caa0f5bee79
Sep 09 14:12:07.302657 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.302599    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-web-config\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.302657 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.302642    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.302874 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.302684    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6527p\" (UniqueName: \"kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-kube-api-access-6527p\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.302874 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.302740    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.302874 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.302768    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.302874 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.302837    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.302874 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.302862    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.302894    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.302943    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.302979    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-config\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.303005    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.303031    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.303056    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.303127    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.303153    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.303180    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.303202    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.303244    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-config-out\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.303269    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.303473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.303299    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.306567 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.304645    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.306567 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.304939    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.306567 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.305276    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.306567 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.305797    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.306567 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.305820    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.306567 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.306151    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.306567 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.306478    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.310212 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.308663    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.310212 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.308744    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.313384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.311389    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.313384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.311762    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.313384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.312591    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.313384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.312867    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.313384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.312996    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-config-out\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.313384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.313036    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.313384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.313336    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-config\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.313384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.313353    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.314801 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.314747    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-web-config\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.316184 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.316132    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6527p\" (UniqueName: \"kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-kube-api-access-6527p\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.319842 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.319806    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.432408 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.432325    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:07.962775 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:07.962672    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 14:12:07.968013 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:07.967979    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536375ad_2f82_46fd_b136_51193afa3b27.slice/crio-e45c9612853885e86f17899344581c7ff575866c9ce1ac555fe9625fac21bbf2 WatchSource:0}: Error finding container e45c9612853885e86f17899344581c7ff575866c9ce1ac555fe9625fac21bbf2: Status 404 returned error can't find the container with id e45c9612853885e86f17899344581c7ff575866c9ce1ac555fe9625fac21bbf2
Sep 09 14:12:08.066592 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.066525    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl" event={"ID":"a68a09d4-7a18-44db-bcbc-3e2e3946c5f7","Type":"ContainerStarted","Data":"fdd9e4e7101a3c860d6dd679be3c3151918b2ca30879ece88237cd5784e6a588"}
Sep 09 14:12:08.072437 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.072184    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerStarted","Data":"3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871"}
Sep 09 14:12:08.072437 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.072257    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerStarted","Data":"ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad"}
Sep 09 14:12:08.072437 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.072299    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerStarted","Data":"2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b"}
Sep 09 14:12:08.072437 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.072337    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerStarted","Data":"416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992"}
Sep 09 14:12:08.072437 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.072376    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerStarted","Data":"2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a"}
Sep 09 14:12:08.074189 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.074091    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c" event={"ID":"b25de1ba-24ef-4c06-8cba-957b49915805","Type":"ContainerStarted","Data":"fc62bf252b1b8b69d7b7daf1a4cfba903e5bc597762d51abd8924caa0f5bee79"}
Sep 09 14:12:08.078074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.077908    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj" event={"ID":"bd26eb7e-40ac-4286-9ff7-6a2dead1d728","Type":"ContainerStarted","Data":"4d49c3b0c628b1b72efe31329826bf66b9d98bdc14e8dd49de29e5b237fd6907"}
Sep 09 14:12:08.078074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.077975    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj" event={"ID":"bd26eb7e-40ac-4286-9ff7-6a2dead1d728","Type":"ContainerStarted","Data":"13b1b7de653c159290579c3965fdb26e5ace9c73e5b68058c69e7c6f3014b551"}
Sep 09 14:12:08.078074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.078013    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj" event={"ID":"bd26eb7e-40ac-4286-9ff7-6a2dead1d728","Type":"ContainerStarted","Data":"5134f715ecc6b02035d54ef67cccc2e8dd57f1f87af11a8cd654ce941ab70acf"}
Sep 09 14:12:08.080894 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.080437    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerStarted","Data":"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"}
Sep 09 14:12:08.080894 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.080501    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerStarted","Data":"e45c9612853885e86f17899344581c7ff575866c9ce1ac555fe9625fac21bbf2"}
Sep 09 14:12:08.998531 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:08.998495    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8b6ld"
Sep 09 14:12:09.089625 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:09.089589    2532 generic.go:356] "Generic (PLEG): container finished" podID="536375ad-2f82-46fd-b136-51193afa3b27" containerID="88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285" exitCode=0
Sep 09 14:12:09.089847 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:09.089711    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerDied","Data":"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"}
Sep 09 14:12:09.858786 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:09.858764    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556794d8df-488p4"]
Sep 09 14:12:09.890194 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:09.889500    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7445cdd67c-r84pz"]
Sep 09 14:12:09.892535 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:09.892429    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:09.913427 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:09.909633    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7445cdd67c-r84pz"]
Sep 09 14:12:10.036760 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.036728    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fmtx\" (UniqueName: \"kubernetes.io/projected/b8fff946-047c-477d-bd89-be5eeeebbd21-kube-api-access-6fmtx\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.037196 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.037174    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-oauth-config\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.037427 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.037400    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-serving-cert\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.037585 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.037567    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-console-config\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.037695 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.037680    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-trusted-ca-bundle\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.037848 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.037827    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-oauth-serving-cert\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.037911 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.037891    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-service-ca\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.097733 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.097394    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj" event={"ID":"bd26eb7e-40ac-4286-9ff7-6a2dead1d728","Type":"ContainerStarted","Data":"98e86ca13d1a0136466a53638c722f920017bc2d7e4eb7235df0aeba5eab7eb4"}
Sep 09 14:12:10.097733 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.097471    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj" event={"ID":"bd26eb7e-40ac-4286-9ff7-6a2dead1d728","Type":"ContainerStarted","Data":"705a89a6b8d445a7cc8030ee29d78f91564da8fe9a55927bdc2f422c58125fcc"}
Sep 09 14:12:10.097733 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.097512    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj" event={"ID":"bd26eb7e-40ac-4286-9ff7-6a2dead1d728","Type":"ContainerStarted","Data":"bd1e251e65dde721a6299dccc6d5832ed05b96fd84fc01195f7f8871a94c2b53"}
Sep 09 14:12:10.097733 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.097650    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:10.099174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.099147    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl" event={"ID":"a68a09d4-7a18-44db-bcbc-3e2e3946c5f7","Type":"ContainerStarted","Data":"f6db6b1b49510c6ba0ae7e9cc7c82c26e32b95d6c0a47458fbbd2a949269a057"}
Sep 09 14:12:10.099493 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.099298    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:10.103395 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.103367    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerStarted","Data":"9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20"}
Sep 09 14:12:10.104877 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.104847    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl"
Sep 09 14:12:10.105153 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.105135    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s" event={"ID":"7a9e9fb2-cb35-4732-b75d-4db5e66b3f27","Type":"ContainerStarted","Data":"7cb281fa9712c28a831b8c509afa1e2640b86a5f40b7d47bab84b3659e308789"}
Sep 09 14:12:10.107133 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.107110    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c" event={"ID":"b25de1ba-24ef-4c06-8cba-957b49915805","Type":"ContainerStarted","Data":"cd6a127c6f7b508936608bcbbb2a1df0456b2ec50ddbc84a12cb7e1743d07e56"}
Sep 09 14:12:10.107218 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.107151    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c" event={"ID":"b25de1ba-24ef-4c06-8cba-957b49915805","Type":"ContainerStarted","Data":"f772e71f6d48f8341a706729b0b7d20b779981df0281f93f8d422ba4f869480a"}
Sep 09 14:12:10.107218 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.107170    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c" event={"ID":"b25de1ba-24ef-4c06-8cba-957b49915805","Type":"ContainerStarted","Data":"74f9d16e2ad7559303e3564c2e7fd53dbbc57d2d2521d971e20b32acedac4059"}
Sep 09 14:12:10.120419 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.120383    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj" podStartSLOduration=2.294824426 podStartE2EDuration="7.120370524s" totalImagesPullingTime="4.825546098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:10.119189241 +0000 UTC m=+63.881984484" watchObservedRunningTime="2026-09-09 14:12:10.120370524 +0000 UTC m=+63.883165764"
Sep 09 14:12:10.138946 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.138927    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-console-config\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.139043 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.138953    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-trusted-ca-bundle\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.139043 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.138978    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-oauth-serving-cert\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.139043 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.139009    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-service-ca\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.139187 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.139048    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fmtx\" (UniqueName: \"kubernetes.io/projected/b8fff946-047c-477d-bd89-be5eeeebbd21-kube-api-access-6fmtx\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.139187 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.139087    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-oauth-config\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.139275 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.139237    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-serving-cert\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.139885 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.139729    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-console-config\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.139974 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.139960    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-oauth-serving-cert\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.140020 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.139978    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-service-ca\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.140313 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.140298    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-trusted-ca-bundle\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.141890 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.141872    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-serving-cert\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.142187 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.142166    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-oauth-config\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.147844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.147762    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.5776483089999997 podStartE2EDuration="9.147747133s" totalImagesPullingTime="5.55565334s" totalInitContainerRuntime="14.445484ms" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:10.145468031 +0000 UTC m=+63.908263314" watchObservedRunningTime="2026-09-09 14:12:10.147747133 +0000 UTC m=+63.910542374"
Sep 09 14:12:10.149641 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.149623    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fmtx\" (UniqueName: \"kubernetes.io/projected/b8fff946-047c-477d-bd89-be5eeeebbd21-kube-api-access-6fmtx\") pod \"console-7445cdd67c-r84pz\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") " pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.164277 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.164229    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s" podStartSLOduration=1.295279915 podStartE2EDuration="5.164212558s" totalImagesPullingTime="3.868932643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:10.162591653 +0000 UTC m=+63.925386894" watchObservedRunningTime="2026-09-09 14:12:10.164212558 +0000 UTC m=+63.927007799"
Sep 09 14:12:10.186758 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.186710    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d859d47f9-4pk4c" podStartSLOduration=1.6761828109999999 podStartE2EDuration="4.186692478s" totalImagesPullingTime="2.510509667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:10.183927091 +0000 UTC m=+63.946722343" watchObservedRunningTime="2026-09-09 14:12:10.186692478 +0000 UTC m=+63.949487720"
Sep 09 14:12:10.199594 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.199556    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jw8nl" podStartSLOduration=2.690011096 podStartE2EDuration="5.199546759s" totalImagesPullingTime="2.509535663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:10.198465186 +0000 UTC m=+63.961260429" watchObservedRunningTime="2026-09-09 14:12:10.199546759 +0000 UTC m=+63.962341999"
Sep 09 14:12:10.217677 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.217658    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:10.500848 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.500826    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7445cdd67c-r84pz"]
Sep 09 14:12:10.503395 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:10.503370    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fff946_047c_477d_bd89_be5eeeebbd21.slice/crio-fa489fb5dd9aee4790f7e89866313796555b0f0ebe10447d1fe774cdfc639eff WatchSource:0}: Error finding container fa489fb5dd9aee4790f7e89866313796555b0f0ebe10447d1fe774cdfc639eff: Status 404 returned error can't find the container with id fa489fb5dd9aee4790f7e89866313796555b0f0ebe10447d1fe774cdfc639eff
Sep 09 14:12:10.855438 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.855376    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7445cdd67c-r84pz"]
Sep 09 14:12:10.882292 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.882270    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bb578c749-bcbj7"]
Sep 09 14:12:10.902638 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.902614    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb578c749-bcbj7"]
Sep 09 14:12:10.902736 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:10.902709    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.051459 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.051424    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-oauth-serving-cert\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.051459 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.051464    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-serving-cert\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.051925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.051489    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz94p\" (UniqueName: \"kubernetes.io/projected/57178976-6453-4747-8448-56e257cce257-kube-api-access-mz94p\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.051925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.051610    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-oauth-config\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.051925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.051656    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-console-config\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.051925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.051702    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-service-ca\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.051925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.051858    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-trusted-ca-bundle\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.117331 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.117182    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445cdd67c-r84pz" event={"ID":"b8fff946-047c-477d-bd89-be5eeeebbd21","Type":"ContainerStarted","Data":"9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd"}
Sep 09 14:12:11.117331 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.117243    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445cdd67c-r84pz" event={"ID":"b8fff946-047c-477d-bd89-be5eeeebbd21","Type":"ContainerStarted","Data":"fa489fb5dd9aee4790f7e89866313796555b0f0ebe10447d1fe774cdfc639eff"}
Sep 09 14:12:11.136631 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.136586    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7445cdd67c-r84pz" podStartSLOduration=2.136570152 podStartE2EDuration="2.136570152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:11.135093519 +0000 UTC m=+64.897888760" watchObservedRunningTime="2026-09-09 14:12:11.136570152 +0000 UTC m=+64.899365394"
Sep 09 14:12:11.152581 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.152550    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-trusted-ca-bundle\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.152726 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.152589    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-oauth-serving-cert\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.152726 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.152615    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-serving-cert\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.152726 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.152641    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz94p\" (UniqueName: \"kubernetes.io/projected/57178976-6453-4747-8448-56e257cce257-kube-api-access-mz94p\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.152726 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.152683    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-oauth-config\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.153495 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.153032    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-console-config\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.153495 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.153113    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-service-ca\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.153495 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.153447    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-oauth-serving-cert\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.154028 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.153774    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-service-ca\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.154028 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.153813    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-console-config\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.154357 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.154332    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-trusted-ca-bundle\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.155748 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.155707    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-serving-cert\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.156953 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.156930    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-oauth-config\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.160570 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.160547    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz94p\" (UniqueName: \"kubernetes.io/projected/57178976-6453-4747-8448-56e257cce257-kube-api-access-mz94p\") pod \"console-bb578c749-bcbj7\" (UID: \"57178976-6453-4747-8448-56e257cce257\") " pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.212592 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.212542    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:11.530054 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.529976    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb578c749-bcbj7"]
Sep 09 14:12:11.533858 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:11.533822    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57178976_6453_4747_8448_56e257cce257.slice/crio-f20aea819f90cc861d879ed1b586bb9c6a74122ee1afc78ffa0035798490fdbd WatchSource:0}: Error finding container f20aea819f90cc861d879ed1b586bb9c6a74122ee1afc78ffa0035798490fdbd: Status 404 returned error can't find the container with id f20aea819f90cc861d879ed1b586bb9c6a74122ee1afc78ffa0035798490fdbd
Sep 09 14:12:11.557811 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.557787    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:12:11.557934 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.557838    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:12:11.560542 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.560519    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 14:12:11.560542 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.560525    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 14:12:11.571132 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.571063    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8d7ec4-9e2a-40c5-b008-3c6890b621fe-metrics-certs\") pod \"network-metrics-daemon-lq49r\" (UID: \"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe\") " pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:12:11.572764 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.572719    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d39c37-e950-4def-86cf-bc268ada8e3f-original-pull-secret\") pod \"global-pull-secret-syncer-8xz8x\" (UID: \"04d39c37-e950-4def-86cf-bc268ada8e3f\") " pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:12:11.658720 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.658686    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbvq\" (UniqueName: \"kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq\") pod \"network-check-target-pqw2f\" (UID: \"be00a0fe-fb34-4722-be6e-6ac18ccaed54\") " pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:12:11.660914 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.660893    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 14:12:11.671237 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.671216    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 14:12:11.681828 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.681801    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbvq\" (UniqueName: \"kubernetes.io/projected/be00a0fe-fb34-4722-be6e-6ac18ccaed54-kube-api-access-gkbvq\") pod \"network-check-target-pqw2f\" (UID: \"be00a0fe-fb34-4722-be6e-6ac18ccaed54\") " pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:12:11.740512 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.740479    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-72t49\""
Sep 09 14:12:11.742590 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.742567    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8xz8x"
Sep 09 14:12:11.748635 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.748565    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:12:11.748753 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.748732    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f5zw2\""
Sep 09 14:12:11.757615 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:11.757584    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq49r"
Sep 09 14:12:12.042897 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:12.042867    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8xz8x"]
Sep 09 14:12:12.073652 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:12.073626    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pqw2f"]
Sep 09 14:12:12.117509 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:12.117437    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lq49r"]
Sep 09 14:12:12.125162 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:12.125129    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb578c749-bcbj7" event={"ID":"57178976-6453-4747-8448-56e257cce257","Type":"ContainerStarted","Data":"228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64"}
Sep 09 14:12:12.125292 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:12.125184    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb578c749-bcbj7" event={"ID":"57178976-6453-4747-8448-56e257cce257","Type":"ContainerStarted","Data":"f20aea819f90cc861d879ed1b586bb9c6a74122ee1afc78ffa0035798490fdbd"}
Sep 09 14:12:12.144312 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:12.144256    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bb578c749-bcbj7" podStartSLOduration=2.144237251 podStartE2EDuration="2.144237251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:12.141661605 +0000 UTC m=+65.904456878" watchObservedRunningTime="2026-09-09 14:12:12.144237251 +0000 UTC m=+65.907032491"
Sep 09 14:12:13.105171 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:13.105138    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d39c37_e950_4def_86cf_bc268ada8e3f.slice/crio-fb96d90a6b098d7586cd99950c66ab832d8633f47c2bb8335a7c0e5a81547857 WatchSource:0}: Error finding container fb96d90a6b098d7586cd99950c66ab832d8633f47c2bb8335a7c0e5a81547857: Status 404 returned error can't find the container with id fb96d90a6b098d7586cd99950c66ab832d8633f47c2bb8335a7c0e5a81547857
Sep 09 14:12:13.105964 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:13.105808    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe00a0fe_fb34_4722_be6e_6ac18ccaed54.slice/crio-0771a2927819a1b5dd9d03b40ca24f2074d4215df80af4095133402af2327d86 WatchSource:0}: Error finding container 0771a2927819a1b5dd9d03b40ca24f2074d4215df80af4095133402af2327d86: Status 404 returned error can't find the container with id 0771a2927819a1b5dd9d03b40ca24f2074d4215df80af4095133402af2327d86
Sep 09 14:12:13.106779 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:12:13.106754    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8d7ec4_9e2a_40c5_b008_3c6890b621fe.slice/crio-081395a2e441b6bd6ce75bac6be2d1fcf8042c75745f36db13e5b5eab3e60e2f WatchSource:0}: Error finding container 081395a2e441b6bd6ce75bac6be2d1fcf8042c75745f36db13e5b5eab3e60e2f: Status 404 returned error can't find the container with id 081395a2e441b6bd6ce75bac6be2d1fcf8042c75745f36db13e5b5eab3e60e2f
Sep 09 14:12:13.131725 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:13.131655    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lq49r" event={"ID":"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe","Type":"ContainerStarted","Data":"081395a2e441b6bd6ce75bac6be2d1fcf8042c75745f36db13e5b5eab3e60e2f"}
Sep 09 14:12:13.133147 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:13.133123    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8xz8x" event={"ID":"04d39c37-e950-4def-86cf-bc268ada8e3f","Type":"ContainerStarted","Data":"fb96d90a6b098d7586cd99950c66ab832d8633f47c2bb8335a7c0e5a81547857"}
Sep 09 14:12:13.134444 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:13.134417    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pqw2f" event={"ID":"be00a0fe-fb34-4722-be6e-6ac18ccaed54","Type":"ContainerStarted","Data":"0771a2927819a1b5dd9d03b40ca24f2074d4215df80af4095133402af2327d86"}
Sep 09 14:12:13.758883 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:13.758848    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:14.145326 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:14.145251    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerStarted","Data":"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"}
Sep 09 14:12:14.145684 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:14.145330    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerStarted","Data":"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"}
Sep 09 14:12:14.145684 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:14.145370    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerStarted","Data":"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"}
Sep 09 14:12:14.145684 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:14.145407    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerStarted","Data":"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"}
Sep 09 14:12:14.145684 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:14.145487    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerStarted","Data":"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"}
Sep 09 14:12:14.145684 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:14.145532    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerStarted","Data":"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"}
Sep 09 14:12:14.175322 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:14.175059    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.104100162 podStartE2EDuration="7.17503898s" totalImagesPullingTime="4.044252877s" totalInitContainerRuntime="26.685941ms" isStatefulPod=false podCreationTimestamp="2026-09-09 14:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:14.173469296 +0000 UTC m=+67.936264559" watchObservedRunningTime="2026-09-09 14:12:14.17503898 +0000 UTC m=+67.937834225"
Sep 09 14:12:15.153126 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:15.152954    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lq49r" event={"ID":"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe","Type":"ContainerStarted","Data":"0989ce18fc93d76bb1dc0d204c41cb30365d9f38634513143c882521e005829d"}
Sep 09 14:12:15.153126 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:15.153016    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lq49r" event={"ID":"9a8d7ec4-9e2a-40c5-b008-3c6890b621fe","Type":"ContainerStarted","Data":"a64bb4d2dab08489bc7d15d41208d97e5966efe7c5d41e302769e347eca17bcb"}
Sep 09 14:12:15.170845 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:15.170390    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lq49r" podStartSLOduration=66.417311814 podStartE2EDuration="1m8.170369255s" totalImagesPullingTime="1.753057441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:15.168930397 +0000 UTC m=+68.931725637" watchObservedRunningTime="2026-09-09 14:12:15.170369255 +0000 UTC m=+68.933164496"
Sep 09 14:12:16.124540 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:16.124507    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-58578cccb4-8jclj"
Sep 09 14:12:17.433236 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:17.433195    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:12:18.172501 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:18.172414    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8xz8x" event={"ID":"04d39c37-e950-4def-86cf-bc268ada8e3f","Type":"ContainerStarted","Data":"4e4d5fc8ca1643b5627c47c72a0502ddccbf159f66646eaf80e89b74a5041fb0"}
Sep 09 14:12:18.173910 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:18.173884    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pqw2f" event={"ID":"be00a0fe-fb34-4722-be6e-6ac18ccaed54","Type":"ContainerStarted","Data":"ad6c9c249469eda44497dc863b8d7ee7855c5fddafa20067ee5bfc7497acf5e4"}
Sep 09 14:12:18.174034 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:18.174017    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:12:18.193255 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:18.193210    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8xz8x" podStartSLOduration=66.552867622 podStartE2EDuration="1m11.193197256s" totalImagesPullingTime="4.640329634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:18.191778741 +0000 UTC m=+71.954573988" watchObservedRunningTime="2026-09-09 14:12:18.193197256 +0000 UTC m=+71.955992495"
Sep 09 14:12:18.209173 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:18.209136    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pqw2f" podStartSLOduration=66.573931149 podStartE2EDuration="1m11.209123612s" totalImagesPullingTime="4.635192463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:12:18.207588303 +0000 UTC m=+71.970383546" watchObservedRunningTime="2026-09-09 14:12:18.209123612 +0000 UTC m=+71.971918849"
Sep 09 14:12:20.218005 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:20.217976    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:21.213509 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:21.213476    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:21.213664 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:21.213538    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:21.217945 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:21.217926    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:22.194124 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:22.194083    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:12:22.249373 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:22.249335    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645b7f5d68-vcx9f"]
Sep 09 14:12:25.624071 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:25.624040    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:25.624438 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:25.624132    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:34.886254 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:34.886140    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-556794d8df-488p4" podUID="ce04d435-e1d7-494d-8908-685c0f03befa" containerName="console" containerID="cri-o://3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4" gracePeriod=15
Sep 09 14:12:35.120969 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.120947    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:35.247541 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.247088    2532 generic.go:356] "Generic (PLEG): container finished" podID="ce04d435-e1d7-494d-8908-685c0f03befa" containerID="3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4" exitCode=0
Sep 09 14:12:35.247541 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.247164    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556794d8df-488p4"
Sep 09 14:12:35.247541 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.247189    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556794d8df-488p4" event={"ID":"ce04d435-e1d7-494d-8908-685c0f03befa","Type":"ContainerDied","Data":"3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4"}
Sep 09 14:12:35.247541 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.247237    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556794d8df-488p4" event={"ID":"ce04d435-e1d7-494d-8908-685c0f03befa","Type":"ContainerDied","Data":"c959bc735da1c941f9e8a2edff8177b88fe5eddec9c6db7ac1a2d13510f6d104"}
Sep 09 14:12:35.247541 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.247264    2532 scope.go:117] "RemoveContainer" containerID="3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4"
Sep 09 14:12:35.255691 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.255671    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-service-ca\") pod \"ce04d435-e1d7-494d-8908-685c0f03befa\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") "
Sep 09 14:12:35.255755 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.255720    2532 scope.go:117] "RemoveContainer" containerID="3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4"
Sep 09 14:12:35.255800 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.255763    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-trusted-ca-bundle\") pod \"ce04d435-e1d7-494d-8908-685c0f03befa\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") "
Sep 09 14:12:35.255800 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.255792    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-oauth-config\") pod \"ce04d435-e1d7-494d-8908-685c0f03befa\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") "
Sep 09 14:12:35.255894 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.255843    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-serving-cert\") pod \"ce04d435-e1d7-494d-8908-685c0f03befa\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") "
Sep 09 14:12:35.255894 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.255876    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-console-config\") pod \"ce04d435-e1d7-494d-8908-685c0f03befa\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") "
Sep 09 14:12:35.255988 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.255908    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-oauth-serving-cert\") pod \"ce04d435-e1d7-494d-8908-685c0f03befa\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") "
Sep 09 14:12:35.256037 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.255996    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwzj8\" (UniqueName: \"kubernetes.io/projected/ce04d435-e1d7-494d-8908-685c0f03befa-kube-api-access-nwzj8\") pod \"ce04d435-e1d7-494d-8908-685c0f03befa\" (UID: \"ce04d435-e1d7-494d-8908-685c0f03befa\") "
Sep 09 14:12:35.256086 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:12:35.256054    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4\": container with ID starting with 3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4 not found: ID does not exist" containerID="3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4"
Sep 09 14:12:35.256198 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.256131    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4"} err="failed to get container status \"3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4\": rpc error: code = NotFound desc = could not find container \"3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4\": container with ID starting with 3516f180e6f8c6af7fc7e36b82578b1968d7942bdfc563ef23296caade7c8de4 not found: ID does not exist"
Sep 09 14:12:35.256528 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.256501    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ce04d435-e1d7-494d-8908-685c0f03befa" (UID: "ce04d435-e1d7-494d-8908-685c0f03befa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:35.256623 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.256526    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ce04d435-e1d7-494d-8908-685c0f03befa" (UID: "ce04d435-e1d7-494d-8908-685c0f03befa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:35.256623 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.256562    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-service-ca" (OuterVolumeSpecName: "service-ca") pod "ce04d435-e1d7-494d-8908-685c0f03befa" (UID: "ce04d435-e1d7-494d-8908-685c0f03befa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:35.256732 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.256640    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-console-config" (OuterVolumeSpecName: "console-config") pod "ce04d435-e1d7-494d-8908-685c0f03befa" (UID: "ce04d435-e1d7-494d-8908-685c0f03befa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:35.258050 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.258028    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ce04d435-e1d7-494d-8908-685c0f03befa" (UID: "ce04d435-e1d7-494d-8908-685c0f03befa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:12:35.258176 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.258149    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ce04d435-e1d7-494d-8908-685c0f03befa" (UID: "ce04d435-e1d7-494d-8908-685c0f03befa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:12:35.258227 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.258191    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce04d435-e1d7-494d-8908-685c0f03befa-kube-api-access-nwzj8" (OuterVolumeSpecName: "kube-api-access-nwzj8") pod "ce04d435-e1d7-494d-8908-685c0f03befa" (UID: "ce04d435-e1d7-494d-8908-685c0f03befa"). InnerVolumeSpecName "kube-api-access-nwzj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:12:35.357313 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.357287    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwzj8\" (UniqueName: \"kubernetes.io/projected/ce04d435-e1d7-494d-8908-685c0f03befa-kube-api-access-nwzj8\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:35.357313 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.357310    2532 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:35.357445 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.357318    2532 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-trusted-ca-bundle\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:35.357445 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.357327    2532 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-oauth-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:35.357445 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.357335    2532 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce04d435-e1d7-494d-8908-685c0f03befa-console-serving-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:35.357445 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.357343    2532 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-console-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:35.357445 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.357351    2532 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce04d435-e1d7-494d-8908-685c0f03befa-oauth-serving-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:35.568536 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.568509    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556794d8df-488p4"]
Sep 09 14:12:35.573290 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:35.573270    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-556794d8df-488p4"]
Sep 09 14:12:36.737358 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:36.737327    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce04d435-e1d7-494d-8908-685c0f03befa" path="/var/lib/kubelet/pods/ce04d435-e1d7-494d-8908-685c0f03befa/volumes"
Sep 09 14:12:37.148754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.148675    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7445cdd67c-r84pz" podUID="b8fff946-047c-477d-bd89-be5eeeebbd21" containerName="console" containerID="cri-o://9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd" gracePeriod=15
Sep 09 14:12:37.418866 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.418805    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:37.572986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.572958    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-service-ca\") pod \"b8fff946-047c-477d-bd89-be5eeeebbd21\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") "
Sep 09 14:12:37.573149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.572992    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-oauth-serving-cert\") pod \"b8fff946-047c-477d-bd89-be5eeeebbd21\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") "
Sep 09 14:12:37.573149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.573021    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fmtx\" (UniqueName: \"kubernetes.io/projected/b8fff946-047c-477d-bd89-be5eeeebbd21-kube-api-access-6fmtx\") pod \"b8fff946-047c-477d-bd89-be5eeeebbd21\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") "
Sep 09 14:12:37.573237 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.573151    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-oauth-config\") pod \"b8fff946-047c-477d-bd89-be5eeeebbd21\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") "
Sep 09 14:12:37.573237 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.573179    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-trusted-ca-bundle\") pod \"b8fff946-047c-477d-bd89-be5eeeebbd21\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") "
Sep 09 14:12:37.573237 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.573205    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-serving-cert\") pod \"b8fff946-047c-477d-bd89-be5eeeebbd21\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") "
Sep 09 14:12:37.573237 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.573233    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-console-config\") pod \"b8fff946-047c-477d-bd89-be5eeeebbd21\" (UID: \"b8fff946-047c-477d-bd89-be5eeeebbd21\") "
Sep 09 14:12:37.573514 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.573399    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-service-ca" (OuterVolumeSpecName: "service-ca") pod "b8fff946-047c-477d-bd89-be5eeeebbd21" (UID: "b8fff946-047c-477d-bd89-be5eeeebbd21"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:37.573593 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.573489    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b8fff946-047c-477d-bd89-be5eeeebbd21" (UID: "b8fff946-047c-477d-bd89-be5eeeebbd21"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:37.573689 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.573665    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b8fff946-047c-477d-bd89-be5eeeebbd21" (UID: "b8fff946-047c-477d-bd89-be5eeeebbd21"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:37.573744 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.573723    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-console-config" (OuterVolumeSpecName: "console-config") pod "b8fff946-047c-477d-bd89-be5eeeebbd21" (UID: "b8fff946-047c-477d-bd89-be5eeeebbd21"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:37.575377 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.575353    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b8fff946-047c-477d-bd89-be5eeeebbd21" (UID: "b8fff946-047c-477d-bd89-be5eeeebbd21"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:12:37.575710 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.575688    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b8fff946-047c-477d-bd89-be5eeeebbd21" (UID: "b8fff946-047c-477d-bd89-be5eeeebbd21"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:12:37.575786 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.575726    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fff946-047c-477d-bd89-be5eeeebbd21-kube-api-access-6fmtx" (OuterVolumeSpecName: "kube-api-access-6fmtx") pod "b8fff946-047c-477d-bd89-be5eeeebbd21" (UID: "b8fff946-047c-477d-bd89-be5eeeebbd21"). InnerVolumeSpecName "kube-api-access-6fmtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:12:37.674048 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.673996    2532 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-oauth-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:37.674048 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.674019    2532 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-trusted-ca-bundle\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:37.674048 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.674028    2532 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8fff946-047c-477d-bd89-be5eeeebbd21-console-serving-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:37.674048 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.674037    2532 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-console-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:37.674048 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.674046    2532 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:37.674264 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.674054    2532 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8fff946-047c-477d-bd89-be5eeeebbd21-oauth-serving-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:37.674264 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:37.674062    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fmtx\" (UniqueName: \"kubernetes.io/projected/b8fff946-047c-477d-bd89-be5eeeebbd21-kube-api-access-6fmtx\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:38.262875 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:38.262841    2532 generic.go:356] "Generic (PLEG): container finished" podID="b8fff946-047c-477d-bd89-be5eeeebbd21" containerID="9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd" exitCode=0
Sep 09 14:12:38.263291 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:38.262882    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445cdd67c-r84pz"
Sep 09 14:12:38.263291 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:38.262920    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445cdd67c-r84pz" event={"ID":"b8fff946-047c-477d-bd89-be5eeeebbd21","Type":"ContainerDied","Data":"9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd"}
Sep 09 14:12:38.263291 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:38.262966    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445cdd67c-r84pz" event={"ID":"b8fff946-047c-477d-bd89-be5eeeebbd21","Type":"ContainerDied","Data":"fa489fb5dd9aee4790f7e89866313796555b0f0ebe10447d1fe774cdfc639eff"}
Sep 09 14:12:38.263291 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:38.262986    2532 scope.go:117] "RemoveContainer" containerID="9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd"
Sep 09 14:12:38.272012 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:38.271996    2532 scope.go:117] "RemoveContainer" containerID="9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd"
Sep 09 14:12:38.272306 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:12:38.272287    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd\": container with ID starting with 9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd not found: ID does not exist" containerID="9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd"
Sep 09 14:12:38.272366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:38.272314    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd"} err="failed to get container status \"9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd\": rpc error: code = NotFound desc = could not find container \"9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd\": container with ID starting with 9b59bf095a6d77d16eb301b4b4b9653140d86cb763def71084329171c1a2e7cd not found: ID does not exist"
Sep 09 14:12:38.283056 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:38.283028    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7445cdd67c-r84pz"]
Sep 09 14:12:38.285210 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:38.285189    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7445cdd67c-r84pz"]
Sep 09 14:12:38.734140 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:38.734114    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8fff946-047c-477d-bd89-be5eeeebbd21" path="/var/lib/kubelet/pods/b8fff946-047c-477d-bd89-be5eeeebbd21/volumes"
Sep 09 14:12:45.627934 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:45.627905    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:45.631600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:45.631576    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f49bd8f8-zx26s"
Sep 09 14:12:47.271654 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.271587    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-645b7f5d68-vcx9f" podUID="5b8da9c6-b99a-48ee-a201-92db914e5482" containerName="console" containerID="cri-o://b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b" gracePeriod=15
Sep 09 14:12:47.553437 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.553410    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:12:47.651618 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.651589    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-console-config\") pod \"5b8da9c6-b99a-48ee-a201-92db914e5482\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") "
Sep 09 14:12:47.651796 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.651640    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-oauth-config\") pod \"5b8da9c6-b99a-48ee-a201-92db914e5482\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") "
Sep 09 14:12:47.651796 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.651672    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w67sx\" (UniqueName: \"kubernetes.io/projected/5b8da9c6-b99a-48ee-a201-92db914e5482-kube-api-access-w67sx\") pod \"5b8da9c6-b99a-48ee-a201-92db914e5482\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") "
Sep 09 14:12:47.651796 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.651697    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-serving-cert\") pod \"5b8da9c6-b99a-48ee-a201-92db914e5482\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") "
Sep 09 14:12:47.651796 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.651726    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-oauth-serving-cert\") pod \"5b8da9c6-b99a-48ee-a201-92db914e5482\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") "
Sep 09 14:12:47.651796 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.651761    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-service-ca\") pod \"5b8da9c6-b99a-48ee-a201-92db914e5482\" (UID: \"5b8da9c6-b99a-48ee-a201-92db914e5482\") "
Sep 09 14:12:47.652086 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.652053    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-console-config" (OuterVolumeSpecName: "console-config") pod "5b8da9c6-b99a-48ee-a201-92db914e5482" (UID: "5b8da9c6-b99a-48ee-a201-92db914e5482"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:47.652184 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.652127    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5b8da9c6-b99a-48ee-a201-92db914e5482" (UID: "5b8da9c6-b99a-48ee-a201-92db914e5482"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:47.652390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.652362    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-service-ca" (OuterVolumeSpecName: "service-ca") pod "5b8da9c6-b99a-48ee-a201-92db914e5482" (UID: "5b8da9c6-b99a-48ee-a201-92db914e5482"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:12:47.653976 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.653948    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8da9c6-b99a-48ee-a201-92db914e5482-kube-api-access-w67sx" (OuterVolumeSpecName: "kube-api-access-w67sx") pod "5b8da9c6-b99a-48ee-a201-92db914e5482" (UID: "5b8da9c6-b99a-48ee-a201-92db914e5482"). InnerVolumeSpecName "kube-api-access-w67sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:12:47.654344 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.654323    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5b8da9c6-b99a-48ee-a201-92db914e5482" (UID: "5b8da9c6-b99a-48ee-a201-92db914e5482"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:12:47.654427 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.654413    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5b8da9c6-b99a-48ee-a201-92db914e5482" (UID: "5b8da9c6-b99a-48ee-a201-92db914e5482"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:12:47.752871 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.752843    2532 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-console-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:47.752871 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.752867    2532 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-oauth-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:47.752871 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.752877    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w67sx\" (UniqueName: \"kubernetes.io/projected/5b8da9c6-b99a-48ee-a201-92db914e5482-kube-api-access-w67sx\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:47.753072 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.752886    2532 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b8da9c6-b99a-48ee-a201-92db914e5482-console-serving-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:47.753072 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.752895    2532 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-oauth-serving-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:47.753072 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:47.752903    2532 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8da9c6-b99a-48ee-a201-92db914e5482-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:12:48.307014 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:48.306982    2532 generic.go:356] "Generic (PLEG): container finished" podID="5b8da9c6-b99a-48ee-a201-92db914e5482" containerID="b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b" exitCode=0
Sep 09 14:12:48.307390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:48.307031    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645b7f5d68-vcx9f"
Sep 09 14:12:48.307390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:48.307031    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645b7f5d68-vcx9f" event={"ID":"5b8da9c6-b99a-48ee-a201-92db914e5482","Type":"ContainerDied","Data":"b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b"}
Sep 09 14:12:48.307390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:48.307158    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645b7f5d68-vcx9f" event={"ID":"5b8da9c6-b99a-48ee-a201-92db914e5482","Type":"ContainerDied","Data":"8f502916ce2d14bf5c330a1133a416e1099789105cbc9d79b4f1e5018d2ab762"}
Sep 09 14:12:48.307390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:48.307187    2532 scope.go:117] "RemoveContainer" containerID="b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b"
Sep 09 14:12:48.315462 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:48.315446    2532 scope.go:117] "RemoveContainer" containerID="b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b"
Sep 09 14:12:48.315729 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:12:48.315713    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b\": container with ID starting with b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b not found: ID does not exist" containerID="b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b"
Sep 09 14:12:48.315795 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:48.315734    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b"} err="failed to get container status \"b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b\": rpc error: code = NotFound desc = could not find container \"b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b\": container with ID starting with b8b313023f3f849e13266d1a9cf92bf6fc1de1e11f30ec669538fd7b808de65b not found: ID does not exist"
Sep 09 14:12:48.327220 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:48.327195    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645b7f5d68-vcx9f"]
Sep 09 14:12:48.332661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:48.332643    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-645b7f5d68-vcx9f"]
Sep 09 14:12:48.733402 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:48.733374    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b8da9c6-b99a-48ee-a201-92db914e5482" path="/var/lib/kubelet/pods/5b8da9c6-b99a-48ee-a201-92db914e5482/volumes"
Sep 09 14:12:49.181238 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:12:49.181178    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pqw2f"
Sep 09 14:13:07.433259 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:07.433235    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:07.452618 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:07.452595    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:08.407575 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:08.407551    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:19.017704 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.017672    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d7568d689-njcch"]
Sep 09 14:13:19.018183 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.017987    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce04d435-e1d7-494d-8908-685c0f03befa" containerName="console"
Sep 09 14:13:19.018183 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.017999    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce04d435-e1d7-494d-8908-685c0f03befa" containerName="console"
Sep 09 14:13:19.018183 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.018016    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8da9c6-b99a-48ee-a201-92db914e5482" containerName="console"
Sep 09 14:13:19.018183 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.018022    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8da9c6-b99a-48ee-a201-92db914e5482" containerName="console"
Sep 09 14:13:19.018183 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.018030    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8fff946-047c-477d-bd89-be5eeeebbd21" containerName="console"
Sep 09 14:13:19.018183 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.018036    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8fff946-047c-477d-bd89-be5eeeebbd21" containerName="console"
Sep 09 14:13:19.018183 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.018088    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce04d435-e1d7-494d-8908-685c0f03befa" containerName="console"
Sep 09 14:13:19.018183 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.018110    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8fff946-047c-477d-bd89-be5eeeebbd21" containerName="console"
Sep 09 14:13:19.018183 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.018117    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8da9c6-b99a-48ee-a201-92db914e5482" containerName="console"
Sep 09 14:13:19.021633 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.021614    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.030735 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.030712    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7568d689-njcch"]
Sep 09 14:13:19.109521 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.109493    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-oauth-serving-cert\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.109612 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.109526    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-console-config\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.109612 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.109551    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-serving-cert\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.109683 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.109627    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-trusted-ca-bundle\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.109717 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.109684    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-oauth-config\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.109753 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.109728    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-service-ca\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.109785 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.109752    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdnrn\" (UniqueName: \"kubernetes.io/projected/8b7606f5-a315-467c-b249-5d486fac19cd-kube-api-access-fdnrn\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.210500 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.210475    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-trusted-ca-bundle\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.210582 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.210504    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-oauth-config\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.210582 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.210541    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-service-ca\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.210713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.210696    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdnrn\" (UniqueName: \"kubernetes.io/projected/8b7606f5-a315-467c-b249-5d486fac19cd-kube-api-access-fdnrn\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.210756 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.210744    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-oauth-serving-cert\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.210802 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.210769    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-console-config\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.210860 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.210805    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-serving-cert\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.211336 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.211313    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-service-ca\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.211428 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.211390    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-trusted-ca-bundle\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.211490 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.211467    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-console-config\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.211490 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.211477    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-oauth-serving-cert\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.212891 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.212870    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-oauth-config\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.213058 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.213040    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-serving-cert\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.218292 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.218275    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdnrn\" (UniqueName: \"kubernetes.io/projected/8b7606f5-a315-467c-b249-5d486fac19cd-kube-api-access-fdnrn\") pod \"console-6d7568d689-njcch\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") " pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.330865 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.330812    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:19.598315 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:19.598292    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7568d689-njcch"]
Sep 09 14:13:19.600944 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:13:19.600918    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7606f5_a315_467c_b249_5d486fac19cd.slice/crio-a2effb5519956ebb5fb631da923b65952824ab0c741b47c4c41c8282a1cbb13a WatchSource:0}: Error finding container a2effb5519956ebb5fb631da923b65952824ab0c741b47c4c41c8282a1cbb13a: Status 404 returned error can't find the container with id a2effb5519956ebb5fb631da923b65952824ab0c741b47c4c41c8282a1cbb13a
Sep 09 14:13:20.449159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:20.449124    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7568d689-njcch" event={"ID":"8b7606f5-a315-467c-b249-5d486fac19cd","Type":"ContainerStarted","Data":"5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1"}
Sep 09 14:13:20.449502 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:20.449200    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7568d689-njcch" event={"ID":"8b7606f5-a315-467c-b249-5d486fac19cd","Type":"ContainerStarted","Data":"a2effb5519956ebb5fb631da923b65952824ab0c741b47c4c41c8282a1cbb13a"}
Sep 09 14:13:20.467754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:20.467710    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d7568d689-njcch" podStartSLOduration=2.467696983 podStartE2EDuration="2.467696983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:13:20.466692348 +0000 UTC m=+134.229487601" watchObservedRunningTime="2026-09-09 14:13:20.467696983 +0000 UTC m=+134.230492223"
Sep 09 14:13:21.195662 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.195629    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 14:13:21.195972 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.195932    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="alertmanager" containerID="cri-o://2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a" gracePeriod=120
Sep 09 14:13:21.196061 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.196018    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy-metric" containerID="cri-o://3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871" gracePeriod=120
Sep 09 14:13:21.196220 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.196061    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="prom-label-proxy" containerID="cri-o://9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20" gracePeriod=120
Sep 09 14:13:21.196220 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.196059    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy-web" containerID="cri-o://2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b" gracePeriod=120
Sep 09 14:13:21.196220 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.196077    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="config-reloader" containerID="cri-o://416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992" gracePeriod=120
Sep 09 14:13:21.196220 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.196138    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy" containerID="cri-o://ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad" gracePeriod=120
Sep 09 14:13:21.456713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.456652    2532 generic.go:356] "Generic (PLEG): container finished" podID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerID="9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20" exitCode=0
Sep 09 14:13:21.456713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.456674    2532 generic.go:356] "Generic (PLEG): container finished" podID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerID="3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871" exitCode=0
Sep 09 14:13:21.456713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.456680    2532 generic.go:356] "Generic (PLEG): container finished" podID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerID="ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad" exitCode=0
Sep 09 14:13:21.456713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.456686    2532 generic.go:356] "Generic (PLEG): container finished" podID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerID="416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992" exitCode=0
Sep 09 14:13:21.456713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.456691    2532 generic.go:356] "Generic (PLEG): container finished" podID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerID="2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a" exitCode=0
Sep 09 14:13:21.457346 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.456734    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerDied","Data":"9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20"}
Sep 09 14:13:21.457346 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.456819    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerDied","Data":"3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871"}
Sep 09 14:13:21.457346 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.456872    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerDied","Data":"ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad"}
Sep 09 14:13:21.457346 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.456917    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerDied","Data":"416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992"}
Sep 09 14:13:21.457346 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:21.456966    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerDied","Data":"2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a"}
Sep 09 14:13:22.435637 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.435613    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.464298 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.464266    2532 generic.go:356] "Generic (PLEG): container finished" podID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerID="2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b" exitCode=0
Sep 09 14:13:22.464658 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.464311    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerDied","Data":"2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b"}
Sep 09 14:13:22.464658 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.464348    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.464658 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.464385    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1922ab8d-b3e6-406d-b87c-36a73a7387dd","Type":"ContainerDied","Data":"f7601769ec1fcf6a3fbd0df70568dbaa715f568b5efe989e8800cf9b2b6ae471"}
Sep 09 14:13:22.464658 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.464433    2532 scope.go:117] "RemoveContainer" containerID="9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20"
Sep 09 14:13:22.474055 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.474032    2532 scope.go:117] "RemoveContainer" containerID="3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871"
Sep 09 14:13:22.481758 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.481740    2532 scope.go:117] "RemoveContainer" containerID="ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad"
Sep 09 14:13:22.489395 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.489376    2532 scope.go:117] "RemoveContainer" containerID="2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b"
Sep 09 14:13:22.496571 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.496556    2532 scope.go:117] "RemoveContainer" containerID="416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992"
Sep 09 14:13:22.503738 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.503719    2532 scope.go:117] "RemoveContainer" containerID="2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a"
Sep 09 14:13:22.510937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.510919    2532 scope.go:117] "RemoveContainer" containerID="8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f"
Sep 09 14:13:22.517549 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.517530    2532 scope.go:117] "RemoveContainer" containerID="9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20"
Sep 09 14:13:22.517838 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:22.517818    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20\": container with ID starting with 9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20 not found: ID does not exist" containerID="9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20"
Sep 09 14:13:22.517918 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.517845    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20"} err="failed to get container status \"9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20\": rpc error: code = NotFound desc = could not find container \"9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20\": container with ID starting with 9943596bb220acafc075c7f314a5516e9be78e784513b539cab1947c6123eb20 not found: ID does not exist"
Sep 09 14:13:22.517918 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.517864    2532 scope.go:117] "RemoveContainer" containerID="3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871"
Sep 09 14:13:22.518141 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:22.518122    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871\": container with ID starting with 3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871 not found: ID does not exist" containerID="3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871"
Sep 09 14:13:22.518200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.518148    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871"} err="failed to get container status \"3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871\": rpc error: code = NotFound desc = could not find container \"3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871\": container with ID starting with 3ebddbdbb2b9c14b06237e692605e77aae1a2a27b4c30bd6124b51ac21a00871 not found: ID does not exist"
Sep 09 14:13:22.518200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.518164    2532 scope.go:117] "RemoveContainer" containerID="ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad"
Sep 09 14:13:22.518411 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:22.518395    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad\": container with ID starting with ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad not found: ID does not exist" containerID="ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad"
Sep 09 14:13:22.518459 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.518417    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad"} err="failed to get container status \"ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad\": rpc error: code = NotFound desc = could not find container \"ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad\": container with ID starting with ab96b881fcf57badb9ad5258e3dbf7f526a55d0a6fc12dd056e3d766792629ad not found: ID does not exist"
Sep 09 14:13:22.518459 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.518432    2532 scope.go:117] "RemoveContainer" containerID="2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b"
Sep 09 14:13:22.518683 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:22.518669    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b\": container with ID starting with 2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b not found: ID does not exist" containerID="2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b"
Sep 09 14:13:22.518732 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.518686    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b"} err="failed to get container status \"2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b\": rpc error: code = NotFound desc = could not find container \"2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b\": container with ID starting with 2533b3e7693a6b57a3c97a69e0868b37a514defc324ddb9c72426529b1de214b not found: ID does not exist"
Sep 09 14:13:22.518732 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.518698    2532 scope.go:117] "RemoveContainer" containerID="416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992"
Sep 09 14:13:22.518932 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:22.518915    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992\": container with ID starting with 416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992 not found: ID does not exist" containerID="416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992"
Sep 09 14:13:22.518973 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.518939    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992"} err="failed to get container status \"416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992\": rpc error: code = NotFound desc = could not find container \"416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992\": container with ID starting with 416895cd2808cad67c6cff64eb7e6eab4252ce8d3502136a89bff07b05c92992 not found: ID does not exist"
Sep 09 14:13:22.518973 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.518954    2532 scope.go:117] "RemoveContainer" containerID="2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a"
Sep 09 14:13:22.519232 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:22.519216    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a\": container with ID starting with 2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a not found: ID does not exist" containerID="2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a"
Sep 09 14:13:22.519282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.519234    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a"} err="failed to get container status \"2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a\": rpc error: code = NotFound desc = could not find container \"2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a\": container with ID starting with 2357d3d948de5e872f8c7fa14b1eea0e6fadf49fa7edf73e6c3e44990e81d87a not found: ID does not exist"
Sep 09 14:13:22.519282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.519246    2532 scope.go:117] "RemoveContainer" containerID="8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f"
Sep 09 14:13:22.519479 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:22.519463    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f\": container with ID starting with 8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f not found: ID does not exist" containerID="8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f"
Sep 09 14:13:22.519519 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.519485    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f"} err="failed to get container status \"8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f\": rpc error: code = NotFound desc = could not find container \"8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f\": container with ID starting with 8fef5c7d0251daa5df2b33f6f35735d370616828cb0b58f0a95f9e1b28641a9f not found: ID does not exist"
Sep 09 14:13:22.532801 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.532780    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-volume\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.532874 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.532835    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-tls-assets\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.532914 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.532874    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-main-tls\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.532914 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.532902    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-trusted-ca-bundle\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.532999 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.532931    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-cluster-tls-config\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.532999 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.532961    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.533272 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.533249    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-web-config\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.533369 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.533303    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.533369 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.533321    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:22.533369 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.533364    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw6gz\" (UniqueName: \"kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-kube-api-access-tw6gz\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.533530 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.533395    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-main-db\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.533530 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.533459    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.533530 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.533487    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-metrics-client-ca\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.533530 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.533517    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-out\") pod \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\" (UID: \"1922ab8d-b3e6-406d-b87c-36a73a7387dd\") "
Sep 09 14:13:22.533846 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.533823    2532 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.534289 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.534057    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:13:22.534289 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.534258    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:22.536334 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.536313    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:13:22.536422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.536315    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:22.536656 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.536631    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-volume" (OuterVolumeSpecName: "config-volume") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:22.536726 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.536634    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-out" (OuterVolumeSpecName: "config-out") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:13:22.536853 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.536822    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:22.536948 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.536927    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:22.537013 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.536983    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:22.537629 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.537612    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-kube-api-access-tw6gz" (OuterVolumeSpecName: "kube-api-access-tw6gz") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "kube-api-access-tw6gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:13:22.540865 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.540834    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:22.547332 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.547287    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-web-config" (OuterVolumeSpecName: "web-config") pod "1922ab8d-b3e6-406d-b87c-36a73a7387dd" (UID: "1922ab8d-b3e6-406d-b87c-36a73a7387dd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:22.634973 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.634947    2532 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.634973 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.634969    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw6gz\" (UniqueName: \"kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-kube-api-access-tw6gz\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.635082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.634980    2532 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-alertmanager-main-db\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.635082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.634988    2532 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.635082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.634997    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1922ab8d-b3e6-406d-b87c-36a73a7387dd-metrics-client-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.635082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.635005    2532 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-out\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.635082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.635013    2532 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.635082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.635021    2532 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1922ab8d-b3e6-406d-b87c-36a73a7387dd-tls-assets\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.635082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.635028    2532 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-main-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.635082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.635035    2532 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-cluster-tls-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.635082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.635043    2532 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.635082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.635051    2532 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1922ab8d-b3e6-406d-b87c-36a73a7387dd-web-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:22.782881 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.782856    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 14:13:22.791196 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.791169    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 14:13:22.815768 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.815716    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 14:13:22.816050 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816036    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="config-reloader"
Sep 09 14:13:22.816124 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816052    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="config-reloader"
Sep 09 14:13:22.816124 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816059    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy-metric"
Sep 09 14:13:22.816124 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816064    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy-metric"
Sep 09 14:13:22.816124 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816073    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy-web"
Sep 09 14:13:22.816124 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816079    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy-web"
Sep 09 14:13:22.816124 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816112    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="prom-label-proxy"
Sep 09 14:13:22.816124 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816122    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="prom-label-proxy"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816132    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="alertmanager"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816138    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="alertmanager"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816145    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="init-config-reloader"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816151    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="init-config-reloader"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816156    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816161    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816221    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy-web"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816230    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy-metric"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816236    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="config-reloader"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816241    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="prom-label-proxy"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816246    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="alertmanager"
Sep 09 14:13:22.816329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.816252    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" containerName="kube-rbac-proxy"
Sep 09 14:13:22.820351 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.820334    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.823182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.822809    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 14:13:22.823182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.822893    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 14:13:22.823182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.822904    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 14:13:22.823182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.822913    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-w7c5z\""
Sep 09 14:13:22.823182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.822952    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 14:13:22.823182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.822911    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 14:13:22.823182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.823041    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 14:13:22.823182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.823050    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 14:13:22.823636 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.823359    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 14:13:22.829284 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.829261    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 14:13:22.833423 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.833250    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 14:13:22.936462 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936435    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936569 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936464    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936569 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936489    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936569 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936512    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936673 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936575    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936673 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936622    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-web-config\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936676    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936692    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936708    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936722    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrgrq\" (UniqueName: \"kubernetes.io/projected/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-kube-api-access-hrgrq\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936857 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936785    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-config-out\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936857 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936828    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:22.936857 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:22.936847    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.037830 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.037778    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.037830 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.037832    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038054 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.037864    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038054 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.037894    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038054 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.037929    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038054 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.037974    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-web-config\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038054 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.038028    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038283 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.038055    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038517 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.038479    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038649 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.038628    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038708 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.038668    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrgrq\" (UniqueName: \"kubernetes.io/projected/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-kube-api-access-hrgrq\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038759 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.038703    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-config-out\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038759 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.038708    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038759 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.038745    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.038915 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.038774    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.039669 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.039640    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.041057 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.041026    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.041173 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.041118    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-web-config\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.041246 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.041175    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.041246 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.041174    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.041357 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.041313    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.041520 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.041501    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.041735 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.041713    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.041817 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.041804    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-config-out\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.042969 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.042951    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.045911 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.045895    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrgrq\" (UniqueName: \"kubernetes.io/projected/a6ecf7ca-7459-4e4c-8377-d3a1696377ed-kube-api-access-hrgrq\") pod \"alertmanager-main-0\" (UID: \"a6ecf7ca-7459-4e4c-8377-d3a1696377ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.136269 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.136227    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 14:13:23.413382 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.413268    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 14:13:23.417577 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:13:23.417549    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ecf7ca_7459_4e4c_8377_d3a1696377ed.slice/crio-e4c16f03e651053a1b6eb5706ab7604e09cb35503486aeeda85c06e8367cf8a0 WatchSource:0}: Error finding container e4c16f03e651053a1b6eb5706ab7604e09cb35503486aeeda85c06e8367cf8a0: Status 404 returned error can't find the container with id e4c16f03e651053a1b6eb5706ab7604e09cb35503486aeeda85c06e8367cf8a0
Sep 09 14:13:23.469297 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:23.469271    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6ecf7ca-7459-4e4c-8377-d3a1696377ed","Type":"ContainerStarted","Data":"e4c16f03e651053a1b6eb5706ab7604e09cb35503486aeeda85c06e8367cf8a0"}
Sep 09 14:13:24.475297 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:24.475268    2532 generic.go:356] "Generic (PLEG): container finished" podID="a6ecf7ca-7459-4e4c-8377-d3a1696377ed" containerID="6d9154541108c8087ff3257f4669e25f1ca2a07475438c4dce0c1d8b040982d4" exitCode=0
Sep 09 14:13:24.475631 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:24.475318    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6ecf7ca-7459-4e4c-8377-d3a1696377ed","Type":"ContainerDied","Data":"6d9154541108c8087ff3257f4669e25f1ca2a07475438c4dce0c1d8b040982d4"}
Sep 09 14:13:24.735691 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:24.735665    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1922ab8d-b3e6-406d-b87c-36a73a7387dd" path="/var/lib/kubelet/pods/1922ab8d-b3e6-406d-b87c-36a73a7387dd/volumes"
Sep 09 14:13:25.423286 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.423248    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 14:13:25.423842 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.423811    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="prometheus" containerID="cri-o://326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc" gracePeriod=600
Sep 09 14:13:25.424198 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.424178    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy" containerID="cri-o://367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178" gracePeriod=600
Sep 09 14:13:25.425067 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.424204    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy-web" containerID="cri-o://1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200" gracePeriod=600
Sep 09 14:13:25.425177 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.424232    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="thanos-sidecar" containerID="cri-o://d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6" gracePeriod=600
Sep 09 14:13:25.425349 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.424261    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="config-reloader" containerID="cri-o://3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de" gracePeriod=600
Sep 09 14:13:25.425349 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.424487    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy-thanos" containerID="cri-o://27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869" gracePeriod=600
Sep 09 14:13:25.491867 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.491834    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6ecf7ca-7459-4e4c-8377-d3a1696377ed","Type":"ContainerStarted","Data":"5cc4d32afaf3af44ce97e23e31955e411b32a9bd84cf37c2bf960f5f70c5aad1"}
Sep 09 14:13:25.492242 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.491959    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6ecf7ca-7459-4e4c-8377-d3a1696377ed","Type":"ContainerStarted","Data":"26b14a8a7d0985fe60a1febd13e91a4044a376b5bd3398bfc397615d222c9651"}
Sep 09 14:13:25.492242 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.492002    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6ecf7ca-7459-4e4c-8377-d3a1696377ed","Type":"ContainerStarted","Data":"002a427b02b36baca12bfab4b742be3e63de56eeb5b012f55daa86691cd9756f"}
Sep 09 14:13:25.492242 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.492045    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6ecf7ca-7459-4e4c-8377-d3a1696377ed","Type":"ContainerStarted","Data":"1c57d92a0e0301ae9bc118602fdb70402a091a07d223d85f4837b2f949352e89"}
Sep 09 14:13:25.492242 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.492085    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6ecf7ca-7459-4e4c-8377-d3a1696377ed","Type":"ContainerStarted","Data":"d381db1e28b3d2ce85863eaae51d04ba49427be3750852f98ebdf0b8d655eb2b"}
Sep 09 14:13:25.492960 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.492938    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6ecf7ca-7459-4e4c-8377-d3a1696377ed","Type":"ContainerStarted","Data":"5a58187c036803826a5ddb8e84c5235a3b595420ba7a95333ba61bcb608b49cb"}
Sep 09 14:13:25.517036 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.516984    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.501425073 podStartE2EDuration="3.516964979s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.539906ms" isStatefulPod=false podCreationTimestamp="2026-09-09 14:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:13:25.514926462 +0000 UTC m=+139.277721703" watchObservedRunningTime="2026-09-09 14:13:25.516964979 +0000 UTC m=+139.279760221"
Sep 09 14:13:25.669877 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.669850    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:25.762591 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762553    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-trusted-ca-bundle\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.762782 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762607    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-tls-assets\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.762782 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762638    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-config-out\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.762782 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762676    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-config\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.762782 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762699    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-kube-rbac-proxy\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.762782 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762729    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-db\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.762782 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762762    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762807    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-metrics-client-certs\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762831    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-web-config\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762861    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-metrics-client-ca\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762891    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-0\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762923    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-tls\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762946    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762962    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:25.763128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.762999    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-grpc-tls\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.763032    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-kubelet-serving-ca-bundle\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.763078    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-serving-certs-ca-bundle\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763587 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.763139    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6527p\" (UniqueName: \"kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-kube-api-access-6527p\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763587 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.763180    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-2\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763587 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.763206    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-thanos-prometheus-http-client-file\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763587 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.763237    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-1\") pod \"536375ad-2f82-46fd-b136-51193afa3b27\" (UID: \"536375ad-2f82-46fd-b136-51193afa3b27\") "
Sep 09 14:13:25.763587 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.763328    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:25.763844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.763591    2532 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.763844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.763613    2532 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-metrics-client-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.764288 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.764259    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:13:25.765421 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.765390    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:25.765920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.765890    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:25.765920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.765905    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:25.766762 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.766720    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:25.766929 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.766892    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:25.767374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.767291    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:25.767478 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.767435    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:25.767645 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.767578    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:25.767723 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.767656    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:25.767839 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.767817    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:13:25.767946 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.767859    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:25.768018 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.767949    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-config-out" (OuterVolumeSpecName: "config-out") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:13:25.768321 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.768300    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:25.768386 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.768357    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:25.769380 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.769357    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-config" (OuterVolumeSpecName: "config") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:25.769465 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.769388    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-kube-api-access-6527p" (OuterVolumeSpecName: "kube-api-access-6527p") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "kube-api-access-6527p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:13:25.778446 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.778420    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-web-config" (OuterVolumeSpecName: "web-config") pod "536375ad-2f82-46fd-b136-51193afa3b27" (UID: "536375ad-2f82-46fd-b136-51193afa3b27"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:25.864806 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864782    2532 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864806 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864803    2532 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864815    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6527p\" (UniqueName: \"kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-kube-api-access-6527p\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864825    2532 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864833    2532 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864841    2532 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864851    2532 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/536375ad-2f82-46fd-b136-51193afa3b27-tls-assets\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864859    2532 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-config-out\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864867    2532 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864874    2532 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-kube-rbac-proxy\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864883    2532 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-db\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864891    2532 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864900    2532 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-metrics-client-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864908    2532 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-web-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864916    2532 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/536375ad-2f82-46fd-b136-51193afa3b27-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.864920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864924    2532 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.865332 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864933    2532 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:25.865332 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:25.864941    2532 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/536375ad-2f82-46fd-b136-51193afa3b27-secret-grpc-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:26.499657 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499628    2532 generic.go:356] "Generic (PLEG): container finished" podID="536375ad-2f82-46fd-b136-51193afa3b27" containerID="27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869" exitCode=0
Sep 09 14:13:26.499657 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499652    2532 generic.go:356] "Generic (PLEG): container finished" podID="536375ad-2f82-46fd-b136-51193afa3b27" containerID="367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178" exitCode=0
Sep 09 14:13:26.499657 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499661    2532 generic.go:356] "Generic (PLEG): container finished" podID="536375ad-2f82-46fd-b136-51193afa3b27" containerID="1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200" exitCode=0
Sep 09 14:13:26.499657 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499667    2532 generic.go:356] "Generic (PLEG): container finished" podID="536375ad-2f82-46fd-b136-51193afa3b27" containerID="d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6" exitCode=0
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499682    2532 generic.go:356] "Generic (PLEG): container finished" podID="536375ad-2f82-46fd-b136-51193afa3b27" containerID="3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de" exitCode=0
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499690    2532 generic.go:356] "Generic (PLEG): container finished" podID="536375ad-2f82-46fd-b136-51193afa3b27" containerID="326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc" exitCode=0
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499704    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499715    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerDied","Data":"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"}
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499781    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerDied","Data":"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"}
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499808    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerDied","Data":"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"}
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499833    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerDied","Data":"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"}
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499857    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerDied","Data":"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"}
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499881    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerDied","Data":"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"}
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499872    2532 scope.go:117] "RemoveContainer" containerID="27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"
Sep 09 14:13:26.500145 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.499916    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"536375ad-2f82-46fd-b136-51193afa3b27","Type":"ContainerDied","Data":"e45c9612853885e86f17899344581c7ff575866c9ce1ac555fe9625fac21bbf2"}
Sep 09 14:13:26.508002 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.507985    2532 scope.go:117] "RemoveContainer" containerID="367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"
Sep 09 14:13:26.515027 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.515009    2532 scope.go:117] "RemoveContainer" containerID="1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"
Sep 09 14:13:26.522583 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.522470    2532 scope.go:117] "RemoveContainer" containerID="d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"
Sep 09 14:13:26.523352 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.523329    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 14:13:26.529632 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.529613    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 14:13:26.529849 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.529835    2532 scope.go:117] "RemoveContainer" containerID="3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"
Sep 09 14:13:26.536576 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.536558    2532 scope.go:117] "RemoveContainer" containerID="326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"
Sep 09 14:13:26.543710 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.543553    2532 scope.go:117] "RemoveContainer" containerID="88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"
Sep 09 14:13:26.550203 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.550182    2532 scope.go:117] "RemoveContainer" containerID="27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"
Sep 09 14:13:26.550510 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:26.550465    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": container with ID starting with 27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869 not found: ID does not exist" containerID="27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"
Sep 09 14:13:26.550510 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.550498    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"} err="failed to get container status \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": rpc error: code = NotFound desc = could not find container \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": container with ID starting with 27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869 not found: ID does not exist"
Sep 09 14:13:26.550614 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.550518    2532 scope.go:117] "RemoveContainer" containerID="367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"
Sep 09 14:13:26.550758 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:26.550739    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": container with ID starting with 367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178 not found: ID does not exist" containerID="367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"
Sep 09 14:13:26.550821 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.550768    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"} err="failed to get container status \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": rpc error: code = NotFound desc = could not find container \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": container with ID starting with 367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178 not found: ID does not exist"
Sep 09 14:13:26.550821 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.550790    2532 scope.go:117] "RemoveContainer" containerID="1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"
Sep 09 14:13:26.551032 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:26.551017    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": container with ID starting with 1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200 not found: ID does not exist" containerID="1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"
Sep 09 14:13:26.551082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.551036    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"} err="failed to get container status \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": rpc error: code = NotFound desc = could not find container \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": container with ID starting with 1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200 not found: ID does not exist"
Sep 09 14:13:26.551082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.551053    2532 scope.go:117] "RemoveContainer" containerID="d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"
Sep 09 14:13:26.551281 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:26.551264    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": container with ID starting with d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6 not found: ID does not exist" containerID="d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"
Sep 09 14:13:26.551318 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.551288    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"} err="failed to get container status \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": rpc error: code = NotFound desc = could not find container \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": container with ID starting with d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6 not found: ID does not exist"
Sep 09 14:13:26.551318 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.551306    2532 scope.go:117] "RemoveContainer" containerID="3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"
Sep 09 14:13:26.551520 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:26.551504    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": container with ID starting with 3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de not found: ID does not exist" containerID="3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"
Sep 09 14:13:26.551566 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.551523    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"} err="failed to get container status \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": rpc error: code = NotFound desc = could not find container \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": container with ID starting with 3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de not found: ID does not exist"
Sep 09 14:13:26.551566 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.551534    2532 scope.go:117] "RemoveContainer" containerID="326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"
Sep 09 14:13:26.551743 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:26.551726    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": container with ID starting with 326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc not found: ID does not exist" containerID="326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"
Sep 09 14:13:26.551788 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.551746    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"} err="failed to get container status \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": rpc error: code = NotFound desc = could not find container \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": container with ID starting with 326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc not found: ID does not exist"
Sep 09 14:13:26.551788 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.551765    2532 scope.go:117] "RemoveContainer" containerID="88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"
Sep 09 14:13:26.551976 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:26.551960    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": container with ID starting with 88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285 not found: ID does not exist" containerID="88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"
Sep 09 14:13:26.552033 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.551984    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"} err="failed to get container status \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": rpc error: code = NotFound desc = could not find container \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": container with ID starting with 88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285 not found: ID does not exist"
Sep 09 14:13:26.552033 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.552003    2532 scope.go:117] "RemoveContainer" containerID="27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"
Sep 09 14:13:26.552300 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.552277    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"} err="failed to get container status \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": rpc error: code = NotFound desc = could not find container \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": container with ID starting with 27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869 not found: ID does not exist"
Sep 09 14:13:26.552300 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.552299    2532 scope.go:117] "RemoveContainer" containerID="367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"
Sep 09 14:13:26.552696 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.552674    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"} err="failed to get container status \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": rpc error: code = NotFound desc = could not find container \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": container with ID starting with 367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178 not found: ID does not exist"
Sep 09 14:13:26.552801 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.552698    2532 scope.go:117] "RemoveContainer" containerID="1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"
Sep 09 14:13:26.553077 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.553050    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"} err="failed to get container status \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": rpc error: code = NotFound desc = could not find container \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": container with ID starting with 1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200 not found: ID does not exist"
Sep 09 14:13:26.553077 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.553077    2532 scope.go:117] "RemoveContainer" containerID="d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"
Sep 09 14:13:26.553446 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.553422    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"} err="failed to get container status \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": rpc error: code = NotFound desc = could not find container \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": container with ID starting with d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6 not found: ID does not exist"
Sep 09 14:13:26.553446 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.553446    2532 scope.go:117] "RemoveContainer" containerID="3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"
Sep 09 14:13:26.553768 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.553748    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"} err="failed to get container status \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": rpc error: code = NotFound desc = could not find container \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": container with ID starting with 3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de not found: ID does not exist"
Sep 09 14:13:26.553768 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.553769    2532 scope.go:117] "RemoveContainer" containerID="326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"
Sep 09 14:13:26.554072 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.554045    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"} err="failed to get container status \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": rpc error: code = NotFound desc = could not find container \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": container with ID starting with 326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc not found: ID does not exist"
Sep 09 14:13:26.554072 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.554074    2532 scope.go:117] "RemoveContainer" containerID="88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"
Sep 09 14:13:26.554374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.554350    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"} err="failed to get container status \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": rpc error: code = NotFound desc = could not find container \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": container with ID starting with 88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285 not found: ID does not exist"
Sep 09 14:13:26.554456 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.554375    2532 scope.go:117] "RemoveContainer" containerID="27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"
Sep 09 14:13:26.554628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.554611    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"} err="failed to get container status \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": rpc error: code = NotFound desc = could not find container \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": container with ID starting with 27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869 not found: ID does not exist"
Sep 09 14:13:26.554677 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.554629    2532 scope.go:117] "RemoveContainer" containerID="367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"
Sep 09 14:13:26.554843 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.554827    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"} err="failed to get container status \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": rpc error: code = NotFound desc = could not find container \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": container with ID starting with 367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178 not found: ID does not exist"
Sep 09 14:13:26.554888 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.554844    2532 scope.go:117] "RemoveContainer" containerID="1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"
Sep 09 14:13:26.554888 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.554851    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 14:13:26.555043 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555025    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"} err="failed to get container status \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": rpc error: code = NotFound desc = could not find container \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": container with ID starting with 1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200 not found: ID does not exist"
Sep 09 14:13:26.555122 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555046    2532 scope.go:117] "RemoveContainer" containerID="d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"
Sep 09 14:13:26.555240 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555222    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="prometheus"
Sep 09 14:13:26.555240 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555241    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="prometheus"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555256    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy-web"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555265    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy-web"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555278    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="init-config-reloader"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555286    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="init-config-reloader"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555286    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"} err="failed to get container status \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": rpc error: code = NotFound desc = could not find container \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": container with ID starting with d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6 not found: ID does not exist"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555298    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="thanos-sidecar"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555301    2532 scope.go:117] "RemoveContainer" containerID="3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555307    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="thanos-sidecar"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555329    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555337    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555350    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy-thanos"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555358    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy-thanos"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555382    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="config-reloader"
Sep 09 14:13:26.555391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555390    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="config-reloader"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555471    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="config-reloader"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555484    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="thanos-sidecar"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555493    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy-thanos"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555511    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy-web"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555520    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="prometheus"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555530    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="536375ad-2f82-46fd-b136-51193afa3b27" containerName="kube-rbac-proxy"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555491    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"} err="failed to get container status \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": rpc error: code = NotFound desc = could not find container \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": container with ID starting with 3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de not found: ID does not exist"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555578    2532 scope.go:117] "RemoveContainer" containerID="326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555819    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"} err="failed to get container status \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": rpc error: code = NotFound desc = could not find container \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": container with ID starting with 326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc not found: ID does not exist"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.555847    2532 scope.go:117] "RemoveContainer" containerID="88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"
Sep 09 14:13:26.556092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.556086    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"} err="failed to get container status \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": rpc error: code = NotFound desc = could not find container \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": container with ID starting with 88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285 not found: ID does not exist"
Sep 09 14:13:26.556639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.556121    2532 scope.go:117] "RemoveContainer" containerID="27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"
Sep 09 14:13:26.556639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.556332    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"} err="failed to get container status \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": rpc error: code = NotFound desc = could not find container \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": container with ID starting with 27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869 not found: ID does not exist"
Sep 09 14:13:26.556639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.556347    2532 scope.go:117] "RemoveContainer" containerID="367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"
Sep 09 14:13:26.556639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.556546    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"} err="failed to get container status \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": rpc error: code = NotFound desc = could not find container \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": container with ID starting with 367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178 not found: ID does not exist"
Sep 09 14:13:26.556639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.556560    2532 scope.go:117] "RemoveContainer" containerID="1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"
Sep 09 14:13:26.556871 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.556821    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"} err="failed to get container status \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": rpc error: code = NotFound desc = could not find container \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": container with ID starting with 1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200 not found: ID does not exist"
Sep 09 14:13:26.556871 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.556835    2532 scope.go:117] "RemoveContainer" containerID="d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"
Sep 09 14:13:26.557162 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.557142    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"} err="failed to get container status \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": rpc error: code = NotFound desc = could not find container \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": container with ID starting with d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6 not found: ID does not exist"
Sep 09 14:13:26.557162 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.557161    2532 scope.go:117] "RemoveContainer" containerID="3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"
Sep 09 14:13:26.557400 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.557380    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"} err="failed to get container status \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": rpc error: code = NotFound desc = could not find container \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": container with ID starting with 3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de not found: ID does not exist"
Sep 09 14:13:26.557456 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.557400    2532 scope.go:117] "RemoveContainer" containerID="326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"
Sep 09 14:13:26.557644 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.557627    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"} err="failed to get container status \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": rpc error: code = NotFound desc = could not find container \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": container with ID starting with 326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc not found: ID does not exist"
Sep 09 14:13:26.557685 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.557645    2532 scope.go:117] "RemoveContainer" containerID="88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"
Sep 09 14:13:26.557894 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.557869    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"} err="failed to get container status \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": rpc error: code = NotFound desc = could not find container \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": container with ID starting with 88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285 not found: ID does not exist"
Sep 09 14:13:26.557894 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.557888    2532 scope.go:117] "RemoveContainer" containerID="27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"
Sep 09 14:13:26.558174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.558146    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"} err="failed to get container status \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": rpc error: code = NotFound desc = could not find container \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": container with ID starting with 27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869 not found: ID does not exist"
Sep 09 14:13:26.558174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.558172    2532 scope.go:117] "RemoveContainer" containerID="367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"
Sep 09 14:13:26.558453 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.558434    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"} err="failed to get container status \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": rpc error: code = NotFound desc = could not find container \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": container with ID starting with 367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178 not found: ID does not exist"
Sep 09 14:13:26.558539 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.558454    2532 scope.go:117] "RemoveContainer" containerID="1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"
Sep 09 14:13:26.558748 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.558722    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"} err="failed to get container status \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": rpc error: code = NotFound desc = could not find container \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": container with ID starting with 1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200 not found: ID does not exist"
Sep 09 14:13:26.558827 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.558751    2532 scope.go:117] "RemoveContainer" containerID="d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"
Sep 09 14:13:26.559062 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.559042    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"} err="failed to get container status \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": rpc error: code = NotFound desc = could not find container \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": container with ID starting with d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6 not found: ID does not exist"
Sep 09 14:13:26.559174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.559063    2532 scope.go:117] "RemoveContainer" containerID="3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"
Sep 09 14:13:26.559318 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.559302    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"} err="failed to get container status \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": rpc error: code = NotFound desc = could not find container \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": container with ID starting with 3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de not found: ID does not exist"
Sep 09 14:13:26.559359 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.559319    2532 scope.go:117] "RemoveContainer" containerID="326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"
Sep 09 14:13:26.559539 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.559522    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"} err="failed to get container status \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": rpc error: code = NotFound desc = could not find container \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": container with ID starting with 326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc not found: ID does not exist"
Sep 09 14:13:26.559587 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.559540    2532 scope.go:117] "RemoveContainer" containerID="88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"
Sep 09 14:13:26.559770 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.559752    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"} err="failed to get container status \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": rpc error: code = NotFound desc = could not find container \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": container with ID starting with 88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285 not found: ID does not exist"
Sep 09 14:13:26.559820 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.559770    2532 scope.go:117] "RemoveContainer" containerID="27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"
Sep 09 14:13:26.560009 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.559990    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869"} err="failed to get container status \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": rpc error: code = NotFound desc = could not find container \"27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869\": container with ID starting with 27674f68cd473c46c9b7aa5ca98dbdd4915fd7aa3b63426be0095d2f2d2e2869 not found: ID does not exist"
Sep 09 14:13:26.560059 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.560009    2532 scope.go:117] "RemoveContainer" containerID="367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"
Sep 09 14:13:26.560059 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.560036    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.560398 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.560219    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178"} err="failed to get container status \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": rpc error: code = NotFound desc = could not find container \"367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178\": container with ID starting with 367b3c794c7995c125db777d5014098d215805b2fa2b02608e0518322b00e178 not found: ID does not exist"
Sep 09 14:13:26.560398 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.560253    2532 scope.go:117] "RemoveContainer" containerID="1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"
Sep 09 14:13:26.560670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.560642    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200"} err="failed to get container status \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": rpc error: code = NotFound desc = could not find container \"1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200\": container with ID starting with 1a000ab03fd4eafe251a5063b9b0e27be8810fb71f739237137c7561ae51e200 not found: ID does not exist"
Sep 09 14:13:26.560714 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.560674    2532 scope.go:117] "RemoveContainer" containerID="d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"
Sep 09 14:13:26.560902 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.560880    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6"} err="failed to get container status \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": rpc error: code = NotFound desc = could not find container \"d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6\": container with ID starting with d80e3138014b5b255d8e4ba9871f12347216f425cee87e2f3334ee85818f69a6 not found: ID does not exist"
Sep 09 14:13:26.560952 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.560904    2532 scope.go:117] "RemoveContainer" containerID="3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"
Sep 09 14:13:26.561144 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.561121    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de"} err="failed to get container status \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": rpc error: code = NotFound desc = could not find container \"3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de\": container with ID starting with 3f50022dc6ae7df00cd9adb014bf39c38bd0eb867aad03eb5760a43dd77577de not found: ID does not exist"
Sep 09 14:13:26.561232 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.561145    2532 scope.go:117] "RemoveContainer" containerID="326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"
Sep 09 14:13:26.561422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.561404    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc"} err="failed to get container status \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": rpc error: code = NotFound desc = could not find container \"326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc\": container with ID starting with 326114902e96c4b43bd7d2f845d7c7cb77dcbc6dab8a26e55af61d9f73c490dc not found: ID does not exist"
Sep 09 14:13:26.561467 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.561423    2532 scope.go:117] "RemoveContainer" containerID="88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"
Sep 09 14:13:26.561656 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.561637    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285"} err="failed to get container status \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": rpc error: code = NotFound desc = could not find container \"88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285\": container with ID starting with 88cabdbe9a1329ac6361335b25003bf25a18b34feccd1e3d9d49cb6a684ce285 not found: ID does not exist"
Sep 09 14:13:26.562296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.562277    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 14:13:26.562391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.562283    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 14:13:26.562568 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.562552    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 14:13:26.562661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.562607    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5np16ljfo8h4j\""
Sep 09 14:13:26.562661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.562654    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 14:13:26.562795 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.562669    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 14:13:26.563074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.562999    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 14:13:26.563074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.563054    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 14:13:26.563074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.563069    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 14:13:26.563074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.563059    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 14:13:26.563074 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.563073    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 14:13:26.563489 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.563454    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 14:13:26.563978 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.563939    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 14:13:26.563978 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.564342    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qrsg5\""
Sep 09 14:13:26.568670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.568649    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 14:13:26.570765 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.570744    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 14:13:26.578941 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.578919    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 14:13:26.669841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.669821    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-config\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.669949 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.669847    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.669949 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.669875    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.669949 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.669913    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670062 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670021    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670132 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670060    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670132 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670091    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670242 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670171    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670297 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670223    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670431 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670402    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670431 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670440    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40cabd6a-194f-472a-a714-1bb0b495f954-config-out\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670457    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670490    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-web-config\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670517    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40cabd6a-194f-472a-a714-1bb0b495f954-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670737 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670646    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670737 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670697    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670737 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670726    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670850 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670743    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdll\" (UniqueName: \"kubernetes.io/projected/40cabd6a-194f-472a-a714-1bb0b495f954-kube-api-access-nrdll\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670850 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670766    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.670850 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.670782    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.732672 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.732651    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536375ad-2f82-46fd-b136-51193afa3b27" path="/var/lib/kubelet/pods/536375ad-2f82-46fd-b136-51193afa3b27/volumes"
Sep 09 14:13:26.771522 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771504    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40cabd6a-194f-472a-a714-1bb0b495f954-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771531    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771551    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771566    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771582    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdll\" (UniqueName: \"kubernetes.io/projected/40cabd6a-194f-472a-a714-1bb0b495f954-kube-api-access-nrdll\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771795 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771611    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771795 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771698    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771795 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771745    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-config\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771795 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771771    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771994 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771818    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771994 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771845    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771994 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771880    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771994 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771917    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.771994 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.771958    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.772270 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.772001    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.772270 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.772029    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.772270 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.772060    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.772270 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.772088    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40cabd6a-194f-472a-a714-1bb0b495f954-config-out\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.772270 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.772139    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.772270 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.772167    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-web-config\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.772270 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.772188    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.772595 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.772312    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.772650 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.772638    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.773021 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.772812    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.773343 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.773318    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.774970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.774942    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.775054 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.774981    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.775413 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.775387    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.775503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.775425    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.775587 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.775567    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.775651 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.775630    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40cabd6a-194f-472a-a714-1bb0b495f954-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.775950 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.775912    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40cabd6a-194f-472a-a714-1bb0b495f954-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.776407 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.776389    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.776746 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.776722    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.777161 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.777136    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-web-config\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.777241 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.777220    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40cabd6a-194f-472a-a714-1bb0b495f954-config-out\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.777436 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.777419    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-config\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.777698 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.777676    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.777956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.777939    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40cabd6a-194f-472a-a714-1bb0b495f954-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.780247 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.780230    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdll\" (UniqueName: \"kubernetes.io/projected/40cabd6a-194f-472a-a714-1bb0b495f954-kube-api-access-nrdll\") pod \"prometheus-k8s-0\" (UID: \"40cabd6a-194f-472a-a714-1bb0b495f954\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:26.872431 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:26.872377    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:27.153990 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:27.153904    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 14:13:27.159091 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:13:27.156974    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cabd6a_194f_472a_a714_1bb0b495f954.slice/crio-f81f8af6ddf6b9f749ed8e7a21477db03044641b32af6f82c8d0cbc29ca324d0 WatchSource:0}: Error finding container f81f8af6ddf6b9f749ed8e7a21477db03044641b32af6f82c8d0cbc29ca324d0: Status 404 returned error can't find the container with id f81f8af6ddf6b9f749ed8e7a21477db03044641b32af6f82c8d0cbc29ca324d0
Sep 09 14:13:27.506353 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:27.506311    2532 generic.go:356] "Generic (PLEG): container finished" podID="40cabd6a-194f-472a-a714-1bb0b495f954" containerID="d4d155a162885ede9ff4df3e228a638dd34e318bbded6a7a1acce4756c3ea09f" exitCode=0
Sep 09 14:13:27.506677 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:27.506378    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40cabd6a-194f-472a-a714-1bb0b495f954","Type":"ContainerDied","Data":"d4d155a162885ede9ff4df3e228a638dd34e318bbded6a7a1acce4756c3ea09f"}
Sep 09 14:13:27.506677 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:27.506462    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40cabd6a-194f-472a-a714-1bb0b495f954","Type":"ContainerStarted","Data":"f81f8af6ddf6b9f749ed8e7a21477db03044641b32af6f82c8d0cbc29ca324d0"}
Sep 09 14:13:28.513953 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:28.513921    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40cabd6a-194f-472a-a714-1bb0b495f954","Type":"ContainerStarted","Data":"f1ab685851a354f410a28f6ff86ad84eda7f267a8b89213f5bc1ae02a62dbc75"}
Sep 09 14:13:28.514296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:28.514006    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40cabd6a-194f-472a-a714-1bb0b495f954","Type":"ContainerStarted","Data":"545db58cc882be7d8b3ec9b921ae4a463f3558a965cf3746527731abab22d243"}
Sep 09 14:13:28.514296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:28.514032    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40cabd6a-194f-472a-a714-1bb0b495f954","Type":"ContainerStarted","Data":"269606daa8fbe435d99f7890c516fff8ad418187fb45a2397d9b15756c476231"}
Sep 09 14:13:28.514296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:28.514057    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40cabd6a-194f-472a-a714-1bb0b495f954","Type":"ContainerStarted","Data":"04c0399723e7ff3bcb1316bb2338c9c89f37a0c086bfec18a644d67ee094364a"}
Sep 09 14:13:28.514296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:28.514080    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40cabd6a-194f-472a-a714-1bb0b495f954","Type":"ContainerStarted","Data":"efd1225f9ab508efe301f51d91d9fe8a07e22957767167e63d32d9e0af10c78a"}
Sep 09 14:13:28.514296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:28.514121    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40cabd6a-194f-472a-a714-1bb0b495f954","Type":"ContainerStarted","Data":"d4a1da2ad16d3e51e5fb9e09eec8d3bad0e9d1d6812e3a426715681b6ce54e12"}
Sep 09 14:13:28.544609 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:28.544552    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.528233229 podStartE2EDuration="2.544530899s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.29767ms" isStatefulPod=false podCreationTimestamp="2026-09-09 14:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:13:28.541695008 +0000 UTC m=+142.304490247" watchObservedRunningTime="2026-09-09 14:13:28.544530899 +0000 UTC m=+142.307326141"
Sep 09 14:13:29.331380 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:29.331339    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:29.331556 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:29.331403    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:29.335979 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:29.335955    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:29.522151 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:29.522131    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:13:29.573596 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:29.573572    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb578c749-bcbj7"]
Sep 09 14:13:31.873200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:31.873161    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:13:54.593166 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.593122    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bb578c749-bcbj7" podUID="57178976-6453-4747-8448-56e257cce257" containerName="console" containerID="cri-o://228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64" gracePeriod=15
Sep 09 14:13:54.828383 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.828362    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:13:54.898131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898048    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-serving-cert\") pod \"57178976-6453-4747-8448-56e257cce257\" (UID: \"57178976-6453-4747-8448-56e257cce257\") "
Sep 09 14:13:54.898131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898083    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-trusted-ca-bundle\") pod \"57178976-6453-4747-8448-56e257cce257\" (UID: \"57178976-6453-4747-8448-56e257cce257\") "
Sep 09 14:13:54.898131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898121    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-oauth-config\") pod \"57178976-6453-4747-8448-56e257cce257\" (UID: \"57178976-6453-4747-8448-56e257cce257\") "
Sep 09 14:13:54.898366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898147    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-console-config\") pod \"57178976-6453-4747-8448-56e257cce257\" (UID: \"57178976-6453-4747-8448-56e257cce257\") "
Sep 09 14:13:54.898366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898174    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz94p\" (UniqueName: \"kubernetes.io/projected/57178976-6453-4747-8448-56e257cce257-kube-api-access-mz94p\") pod \"57178976-6453-4747-8448-56e257cce257\" (UID: \"57178976-6453-4747-8448-56e257cce257\") "
Sep 09 14:13:54.898366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898194    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-oauth-serving-cert\") pod \"57178976-6453-4747-8448-56e257cce257\" (UID: \"57178976-6453-4747-8448-56e257cce257\") "
Sep 09 14:13:54.898366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898227    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-service-ca\") pod \"57178976-6453-4747-8448-56e257cce257\" (UID: \"57178976-6453-4747-8448-56e257cce257\") "
Sep 09 14:13:54.898637 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898612    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-console-config" (OuterVolumeSpecName: "console-config") pod "57178976-6453-4747-8448-56e257cce257" (UID: "57178976-6453-4747-8448-56e257cce257"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:54.898728 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898609    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "57178976-6453-4747-8448-56e257cce257" (UID: "57178976-6453-4747-8448-56e257cce257"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:54.898813 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898777    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-service-ca" (OuterVolumeSpecName: "service-ca") pod "57178976-6453-4747-8448-56e257cce257" (UID: "57178976-6453-4747-8448-56e257cce257"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:54.898910 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.898853    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "57178976-6453-4747-8448-56e257cce257" (UID: "57178976-6453-4747-8448-56e257cce257"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:13:54.900358 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.900331    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57178976-6453-4747-8448-56e257cce257-kube-api-access-mz94p" (OuterVolumeSpecName: "kube-api-access-mz94p") pod "57178976-6453-4747-8448-56e257cce257" (UID: "57178976-6453-4747-8448-56e257cce257"). InnerVolumeSpecName "kube-api-access-mz94p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:13:54.900358 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.900350    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "57178976-6453-4747-8448-56e257cce257" (UID: "57178976-6453-4747-8448-56e257cce257"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:54.900479 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.900402    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "57178976-6453-4747-8448-56e257cce257" (UID: "57178976-6453-4747-8448-56e257cce257"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:13:54.999400 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.999378    2532 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-console-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:54.999400 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.999399    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz94p\" (UniqueName: \"kubernetes.io/projected/57178976-6453-4747-8448-56e257cce257-kube-api-access-mz94p\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:54.999522 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.999409    2532 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-oauth-serving-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:54.999522 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.999418    2532 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:54.999522 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.999426    2532 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-serving-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:54.999522 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.999435    2532 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57178976-6453-4747-8448-56e257cce257-trusted-ca-bundle\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:54.999522 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:54.999443    2532 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57178976-6453-4747-8448-56e257cce257-console-oauth-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:13:55.626632 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:55.626593    2532 generic.go:356] "Generic (PLEG): container finished" podID="57178976-6453-4747-8448-56e257cce257" containerID="228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64" exitCode=0
Sep 09 14:13:55.626632 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:55.626635    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb578c749-bcbj7" event={"ID":"57178976-6453-4747-8448-56e257cce257","Type":"ContainerDied","Data":"228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64"}
Sep 09 14:13:55.627224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:55.626662    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb578c749-bcbj7"
Sep 09 14:13:55.627224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:55.626705    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb578c749-bcbj7" event={"ID":"57178976-6453-4747-8448-56e257cce257","Type":"ContainerDied","Data":"f20aea819f90cc861d879ed1b586bb9c6a74122ee1afc78ffa0035798490fdbd"}
Sep 09 14:13:55.627224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:55.626728    2532 scope.go:117] "RemoveContainer" containerID="228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64"
Sep 09 14:13:55.635297 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:55.635242    2532 scope.go:117] "RemoveContainer" containerID="228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64"
Sep 09 14:13:55.635526 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:13:55.635507    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64\": container with ID starting with 228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64 not found: ID does not exist" containerID="228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64"
Sep 09 14:13:55.635588 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:55.635531    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64"} err="failed to get container status \"228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64\": rpc error: code = NotFound desc = could not find container \"228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64\": container with ID starting with 228e0b3f79182118505c09dc0a4965fb6f47526e8280fa338eef40a0b7277b64 not found: ID does not exist"
Sep 09 14:13:55.647325 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:55.647300    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb578c749-bcbj7"]
Sep 09 14:13:55.648675 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:55.648656    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bb578c749-bcbj7"]
Sep 09 14:13:56.733761 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:13:56.733726    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57178976-6453-4747-8448-56e257cce257" path="/var/lib/kubelet/pods/57178976-6453-4747-8448-56e257cce257/volumes"
Sep 09 14:14:26.872971 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:14:26.872932    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:14:26.888371 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:14:26.888346    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:14:27.762425 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:14:27.762399    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 14:15:11.139897 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.139864    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-s5rw4"]
Sep 09 14:15:11.140336 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.140188    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57178976-6453-4747-8448-56e257cce257" containerName="console"
Sep 09 14:15:11.140336 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.140199    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="57178976-6453-4747-8448-56e257cce257" containerName="console"
Sep 09 14:15:11.140336 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.140280    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="57178976-6453-4747-8448-56e257cce257" containerName="console"
Sep 09 14:15:11.143270 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.143253    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4"
Sep 09 14:15:11.145838 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.145815    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 14:15:11.146623 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.146606    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2rfx7\""
Sep 09 14:15:11.146681 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.146608    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 14:15:11.153366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.153344    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-s5rw4"]
Sep 09 14:15:11.269327 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.269302    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfmqz\" (UniqueName: \"kubernetes.io/projected/9e797538-e161-4a0e-a6bc-acb60dc50097-kube-api-access-lfmqz\") pod \"cert-manager-webhook-6486ffd997-s5rw4\" (UID: \"9e797538-e161-4a0e-a6bc-acb60dc50097\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4"
Sep 09 14:15:11.269434 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.269341    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e797538-e161-4a0e-a6bc-acb60dc50097-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-s5rw4\" (UID: \"9e797538-e161-4a0e-a6bc-acb60dc50097\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4"
Sep 09 14:15:11.369906 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.369884    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmqz\" (UniqueName: \"kubernetes.io/projected/9e797538-e161-4a0e-a6bc-acb60dc50097-kube-api-access-lfmqz\") pod \"cert-manager-webhook-6486ffd997-s5rw4\" (UID: \"9e797538-e161-4a0e-a6bc-acb60dc50097\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4"
Sep 09 14:15:11.370012 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.369917    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e797538-e161-4a0e-a6bc-acb60dc50097-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-s5rw4\" (UID: \"9e797538-e161-4a0e-a6bc-acb60dc50097\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4"
Sep 09 14:15:11.377593 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.377573    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfmqz\" (UniqueName: \"kubernetes.io/projected/9e797538-e161-4a0e-a6bc-acb60dc50097-kube-api-access-lfmqz\") pod \"cert-manager-webhook-6486ffd997-s5rw4\" (UID: \"9e797538-e161-4a0e-a6bc-acb60dc50097\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4"
Sep 09 14:15:11.377849 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.377830    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e797538-e161-4a0e-a6bc-acb60dc50097-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-s5rw4\" (UID: \"9e797538-e161-4a0e-a6bc-acb60dc50097\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4"
Sep 09 14:15:11.466546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.466496    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4"
Sep 09 14:15:11.732071 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.732045    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-s5rw4"]
Sep 09 14:15:11.736268 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:15:11.736238    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e797538_e161_4a0e_a6bc_acb60dc50097.slice/crio-c69a2df7179eec255283b25fd9225edfb4373851af17b1366d20cb55c734a736 WatchSource:0}: Error finding container c69a2df7179eec255283b25fd9225edfb4373851af17b1366d20cb55c734a736: Status 404 returned error can't find the container with id c69a2df7179eec255283b25fd9225edfb4373851af17b1366d20cb55c734a736
Sep 09 14:15:11.913692 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:11.913663    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4" event={"ID":"9e797538-e161-4a0e-a6bc-acb60dc50097","Type":"ContainerStarted","Data":"c69a2df7179eec255283b25fd9225edfb4373851af17b1366d20cb55c734a736"}
Sep 09 14:15:15.935482 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:15.935445    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4" event={"ID":"9e797538-e161-4a0e-a6bc-acb60dc50097","Type":"ContainerStarted","Data":"6ff37c1001bf9f0793ece233cb86b78589800922c0249315230a211cad850296"}
Sep 09 14:15:15.935941 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:15.935562    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4"
Sep 09 14:15:15.951808 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:15.951760    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4" podStartSLOduration=1.7112948220000002 podStartE2EDuration="4.951747631s" totalImagesPullingTime="3.240452809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:15:15.95013489 +0000 UTC m=+249.712930133" watchObservedRunningTime="2026-09-09 14:15:15.951747631 +0000 UTC m=+249.714542871"
Sep 09 14:15:16.018137 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.018113    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-92t74"]
Sep 09 14:15:16.021509 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.021494    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-92t74"
Sep 09 14:15:16.023832 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.023816    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wtq5p\""
Sep 09 14:15:16.031872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.031852    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-92t74"]
Sep 09 14:15:16.110418 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.110393    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68h8\" (UniqueName: \"kubernetes.io/projected/c302cff5-b9ef-4831-b197-31ea5ed15647-kube-api-access-q68h8\") pod \"cert-manager-55b4d954d5-92t74\" (UID: \"c302cff5-b9ef-4831-b197-31ea5ed15647\") " pod="cert-manager/cert-manager-55b4d954d5-92t74"
Sep 09 14:15:16.110514 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.110420    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c302cff5-b9ef-4831-b197-31ea5ed15647-bound-sa-token\") pod \"cert-manager-55b4d954d5-92t74\" (UID: \"c302cff5-b9ef-4831-b197-31ea5ed15647\") " pod="cert-manager/cert-manager-55b4d954d5-92t74"
Sep 09 14:15:16.211245 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.211186    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68h8\" (UniqueName: \"kubernetes.io/projected/c302cff5-b9ef-4831-b197-31ea5ed15647-kube-api-access-q68h8\") pod \"cert-manager-55b4d954d5-92t74\" (UID: \"c302cff5-b9ef-4831-b197-31ea5ed15647\") " pod="cert-manager/cert-manager-55b4d954d5-92t74"
Sep 09 14:15:16.211245 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.211218    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c302cff5-b9ef-4831-b197-31ea5ed15647-bound-sa-token\") pod \"cert-manager-55b4d954d5-92t74\" (UID: \"c302cff5-b9ef-4831-b197-31ea5ed15647\") " pod="cert-manager/cert-manager-55b4d954d5-92t74"
Sep 09 14:15:16.219740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.219715    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c302cff5-b9ef-4831-b197-31ea5ed15647-bound-sa-token\") pod \"cert-manager-55b4d954d5-92t74\" (UID: \"c302cff5-b9ef-4831-b197-31ea5ed15647\") " pod="cert-manager/cert-manager-55b4d954d5-92t74"
Sep 09 14:15:16.219820 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.219784    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68h8\" (UniqueName: \"kubernetes.io/projected/c302cff5-b9ef-4831-b197-31ea5ed15647-kube-api-access-q68h8\") pod \"cert-manager-55b4d954d5-92t74\" (UID: \"c302cff5-b9ef-4831-b197-31ea5ed15647\") " pod="cert-manager/cert-manager-55b4d954d5-92t74"
Sep 09 14:15:16.329918 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.329898    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-92t74"
Sep 09 14:15:16.597441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.597414    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-92t74"]
Sep 09 14:15:16.600268 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:15:16.600244    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc302cff5_b9ef_4831_b197_31ea5ed15647.slice/crio-963343efb607684b750c66b6d2288e683bfb60ec9a5a31a2be31a72bb79940f1 WatchSource:0}: Error finding container 963343efb607684b750c66b6d2288e683bfb60ec9a5a31a2be31a72bb79940f1: Status 404 returned error can't find the container with id 963343efb607684b750c66b6d2288e683bfb60ec9a5a31a2be31a72bb79940f1
Sep 09 14:15:16.940498 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.940417    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-92t74" event={"ID":"c302cff5-b9ef-4831-b197-31ea5ed15647","Type":"ContainerStarted","Data":"469d6ff8cbb7ba978c64125559077ead330efebec94b0bde2448642e23c7ba43"}
Sep 09 14:15:16.940498 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.940470    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-92t74" event={"ID":"c302cff5-b9ef-4831-b197-31ea5ed15647","Type":"ContainerStarted","Data":"963343efb607684b750c66b6d2288e683bfb60ec9a5a31a2be31a72bb79940f1"}
Sep 09 14:15:16.956966 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:16.956926    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-92t74" podStartSLOduration=0.956910907 podStartE2EDuration="956.910907ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:15:16.955806361 +0000 UTC m=+250.718601603" watchObservedRunningTime="2026-09-09 14:15:16.956910907 +0000 UTC m=+250.719706148"
Sep 09 14:15:21.942529 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:21.942499    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-s5rw4"
Sep 09 14:15:49.466852 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.466774    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"]
Sep 09 14:15:49.470922 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.470904    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.473161 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.473139    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 14:15:49.473256 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.473143    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 14:15:49.474090 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.474061    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 14:15:49.474090 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.474079    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 14:15:49.474247 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.474109    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vtcw9\""
Sep 09 14:15:49.474247 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.474157    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 14:15:49.478772 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.478386    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"]
Sep 09 14:15:49.568717 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.568686    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6v2\" (UniqueName: \"kubernetes.io/projected/5f27ee2a-4c63-4595-850f-bb657c264115-kube-api-access-xq6v2\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.568841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.568723    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5f27ee2a-4c63-4595-850f-bb657c264115-manager-config\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.568841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.568744    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f27ee2a-4c63-4595-850f-bb657c264115-metrics-cert\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.568841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.568831    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f27ee2a-4c63-4595-850f-bb657c264115-cert\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.669727 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.669696    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6v2\" (UniqueName: \"kubernetes.io/projected/5f27ee2a-4c63-4595-850f-bb657c264115-kube-api-access-xq6v2\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.669844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.669736    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5f27ee2a-4c63-4595-850f-bb657c264115-manager-config\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.669844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.669768    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f27ee2a-4c63-4595-850f-bb657c264115-metrics-cert\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.669844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.669822    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f27ee2a-4c63-4595-850f-bb657c264115-cert\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.670330 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.670310    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5f27ee2a-4c63-4595-850f-bb657c264115-manager-config\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.672184 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.672164    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f27ee2a-4c63-4595-850f-bb657c264115-metrics-cert\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.672265 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.672218    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f27ee2a-4c63-4595-850f-bb657c264115-cert\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.683477 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.683455    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6v2\" (UniqueName: \"kubernetes.io/projected/5f27ee2a-4c63-4595-850f-bb657c264115-kube-api-access-xq6v2\") pod \"lws-controller-manager-784f875f86-7g92d\" (UID: \"5f27ee2a-4c63-4595-850f-bb657c264115\") " pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:49.781194 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:49.781173    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:50.051461 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:50.051354    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"]
Sep 09 14:15:50.057431 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:15:50.057408    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f27ee2a_4c63_4595_850f_bb657c264115.slice/crio-3f150303be0100b2ba2b99fa764f0028c6b7920edb626c5febfd21ec48c337d5 WatchSource:0}: Error finding container 3f150303be0100b2ba2b99fa764f0028c6b7920edb626c5febfd21ec48c337d5: Status 404 returned error can't find the container with id 3f150303be0100b2ba2b99fa764f0028c6b7920edb626c5febfd21ec48c337d5
Sep 09 14:15:50.070570 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:50.070542    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d" event={"ID":"5f27ee2a-4c63-4595-850f-bb657c264115","Type":"ContainerStarted","Data":"3f150303be0100b2ba2b99fa764f0028c6b7920edb626c5febfd21ec48c337d5"}
Sep 09 14:15:53.086426 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:53.086387    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d" event={"ID":"5f27ee2a-4c63-4595-850f-bb657c264115","Type":"ContainerStarted","Data":"43073bd6eef0ac77ae58ca18d4dfb39f599282b539ce0b82b6b980258e29356b"}
Sep 09 14:15:53.086842 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:53.086503    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:15:53.103663 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:53.103614    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d" podStartSLOduration=1.796855736 podStartE2EDuration="4.103600464s" totalImagesPullingTime="2.306744728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:15:53.102789251 +0000 UTC m=+286.865584503" watchObservedRunningTime="2026-09-09 14:15:53.103600464 +0000 UTC m=+286.866395704"
Sep 09 14:15:54.350070 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.350036    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"]
Sep 09 14:15:54.352561 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.352544    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.355420 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.355402    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 14:15:54.355729 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.355707    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 14:15:54.355793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.355713    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 14:15:54.355793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.355733    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 14:15:54.356177 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.356161    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6nlw9\""
Sep 09 14:15:54.366952 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.366934    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"]
Sep 09 14:15:54.412880 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.412851    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdb45\" (UniqueName: \"kubernetes.io/projected/0c159d6b-25ae-4b67-8596-64198ca7e225-kube-api-access-mdb45\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.412986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.412906    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c159d6b-25ae-4b67-8596-64198ca7e225-apiservice-cert\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.412986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.412932    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c159d6b-25ae-4b67-8596-64198ca7e225-webhook-cert\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.412986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.412953    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c159d6b-25ae-4b67-8596-64198ca7e225-tmp\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.412986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.412974    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c159d6b-25ae-4b67-8596-64198ca7e225-manifests\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.513834 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.513808    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c159d6b-25ae-4b67-8596-64198ca7e225-apiservice-cert\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.513915 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.513839    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c159d6b-25ae-4b67-8596-64198ca7e225-webhook-cert\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.513915 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.513862    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c159d6b-25ae-4b67-8596-64198ca7e225-tmp\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.513915 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.513891    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c159d6b-25ae-4b67-8596-64198ca7e225-manifests\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.514025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.513934    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdb45\" (UniqueName: \"kubernetes.io/projected/0c159d6b-25ae-4b67-8596-64198ca7e225-kube-api-access-mdb45\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.514370 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.514346    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c159d6b-25ae-4b67-8596-64198ca7e225-manifests\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.516112 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.516076    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c159d6b-25ae-4b67-8596-64198ca7e225-tmp\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.516330 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.516311    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c159d6b-25ae-4b67-8596-64198ca7e225-apiservice-cert\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.516373 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.516350    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c159d6b-25ae-4b67-8596-64198ca7e225-webhook-cert\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.521617 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.521594    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdb45\" (UniqueName: \"kubernetes.io/projected/0c159d6b-25ae-4b67-8596-64198ca7e225-kube-api-access-mdb45\") pod \"opendatahub-operator-controller-manager-686db84dc8-wkg7l\" (UID: \"0c159d6b-25ae-4b67-8596-64198ca7e225\") " pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.663441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.663392    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:15:54.943605 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:54.943585    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"]
Sep 09 14:15:54.946301 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:15:54.946270    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c159d6b_25ae_4b67_8596_64198ca7e225.slice/crio-d38d0245d0214402766795d6811fde85e24f92bb200d73a97cc1907a6351c0fa WatchSource:0}: Error finding container d38d0245d0214402766795d6811fde85e24f92bb200d73a97cc1907a6351c0fa: Status 404 returned error can't find the container with id d38d0245d0214402766795d6811fde85e24f92bb200d73a97cc1907a6351c0fa
Sep 09 14:15:55.096373 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:55.096334    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l" event={"ID":"0c159d6b-25ae-4b67-8596-64198ca7e225","Type":"ContainerStarted","Data":"d38d0245d0214402766795d6811fde85e24f92bb200d73a97cc1907a6351c0fa"}
Sep 09 14:15:58.112988 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:58.112956    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l" event={"ID":"0c159d6b-25ae-4b67-8596-64198ca7e225","Type":"ContainerStarted","Data":"94c2219efb5f751f2aa2f4ab039527f28b5d3f5857fd8c468e6a6809bc86d436"}
Sep 09 14:15:59.118655 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:59.118622    2532 generic.go:356] "Generic (PLEG): container finished" podID="0c159d6b-25ae-4b67-8596-64198ca7e225" containerID="94c2219efb5f751f2aa2f4ab039527f28b5d3f5857fd8c468e6a6809bc86d436" exitCode=0
Sep 09 14:15:59.118953 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:15:59.118678    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l" event={"ID":"0c159d6b-25ae-4b67-8596-64198ca7e225","Type":"ContainerDied","Data":"94c2219efb5f751f2aa2f4ab039527f28b5d3f5857fd8c468e6a6809bc86d436"}
Sep 09 14:16:00.125142 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:00.125081    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l" event={"ID":"0c159d6b-25ae-4b67-8596-64198ca7e225","Type":"ContainerStarted","Data":"51eb30c2dee39d22b60809baa248044d4df33428b5f07467e2d48b931a681e4e"}
Sep 09 14:16:00.125647 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:00.125229    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:16:00.147493 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:00.147433    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l" podStartSLOduration=1.794023068 podStartE2EDuration="6.147414445s" totalImagesPullingTime="3.296659601s" totalInitContainerRuntime="1.056731776s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:15:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:16:00.145388825 +0000 UTC m=+293.908184092" watchObservedRunningTime="2026-09-09 14:16:00.147414445 +0000 UTC m=+293.910209686"
Sep 09 14:16:04.092536 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:04.092505    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-784f875f86-7g92d"
Sep 09 14:16:06.625969 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:06.625934    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:16:06.627871 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:06.627851    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:16:06.632367 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:06.632348    2532 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 14:16:11.131211 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:11.131181    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-686db84dc8-wkg7l"
Sep 09 14:16:13.809676 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.809649    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"]
Sep 09 14:16:13.815747 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.815729    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:13.818735 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.818717    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 14:16:13.819109 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.819082    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 14:16:13.819163 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.819127    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 14:16:13.819163 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.819145    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xhdmz\""
Sep 09 14:16:13.819243 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.819170    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 14:16:13.834352 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.834327    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"]
Sep 09 14:16:13.853784 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.853758    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8938794-daba-4785-b9d2-5b8d9870e6a8-tls-certs\") pod \"kube-auth-proxy-6dd84d6f45-t56sb\" (UID: \"a8938794-daba-4785-b9d2-5b8d9870e6a8\") " pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:13.853880 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.853793    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8938794-daba-4785-b9d2-5b8d9870e6a8-tmp\") pod \"kube-auth-proxy-6dd84d6f45-t56sb\" (UID: \"a8938794-daba-4785-b9d2-5b8d9870e6a8\") " pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:13.853880 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.853835    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pffn\" (UniqueName: \"kubernetes.io/projected/a8938794-daba-4785-b9d2-5b8d9870e6a8-kube-api-access-7pffn\") pod \"kube-auth-proxy-6dd84d6f45-t56sb\" (UID: \"a8938794-daba-4785-b9d2-5b8d9870e6a8\") " pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:13.954958 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.954927    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8938794-daba-4785-b9d2-5b8d9870e6a8-tmp\") pod \"kube-auth-proxy-6dd84d6f45-t56sb\" (UID: \"a8938794-daba-4785-b9d2-5b8d9870e6a8\") " pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:13.955077 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.954994    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pffn\" (UniqueName: \"kubernetes.io/projected/a8938794-daba-4785-b9d2-5b8d9870e6a8-kube-api-access-7pffn\") pod \"kube-auth-proxy-6dd84d6f45-t56sb\" (UID: \"a8938794-daba-4785-b9d2-5b8d9870e6a8\") " pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:13.955077 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.955061    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8938794-daba-4785-b9d2-5b8d9870e6a8-tls-certs\") pod \"kube-auth-proxy-6dd84d6f45-t56sb\" (UID: \"a8938794-daba-4785-b9d2-5b8d9870e6a8\") " pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:13.957061 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.957038    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8938794-daba-4785-b9d2-5b8d9870e6a8-tmp\") pod \"kube-auth-proxy-6dd84d6f45-t56sb\" (UID: \"a8938794-daba-4785-b9d2-5b8d9870e6a8\") " pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:13.957199 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.957180    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8938794-daba-4785-b9d2-5b8d9870e6a8-tls-certs\") pod \"kube-auth-proxy-6dd84d6f45-t56sb\" (UID: \"a8938794-daba-4785-b9d2-5b8d9870e6a8\") " pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:13.967449 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:13.967425    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pffn\" (UniqueName: \"kubernetes.io/projected/a8938794-daba-4785-b9d2-5b8d9870e6a8-kube-api-access-7pffn\") pod \"kube-auth-proxy-6dd84d6f45-t56sb\" (UID: \"a8938794-daba-4785-b9d2-5b8d9870e6a8\") " pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:14.125077 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:14.125012    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"
Sep 09 14:16:14.393441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:14.393417    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb"]
Sep 09 14:16:14.395715 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:16:14.395693    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8938794_daba_4785_b9d2_5b8d9870e6a8.slice/crio-e13b13f079f71ef36d8e17dfa5e89fdceaf47dd714c618b127f660f7164d539e WatchSource:0}: Error finding container e13b13f079f71ef36d8e17dfa5e89fdceaf47dd714c618b127f660f7164d539e: Status 404 returned error can't find the container with id e13b13f079f71ef36d8e17dfa5e89fdceaf47dd714c618b127f660f7164d539e
Sep 09 14:16:14.397657 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:14.397642    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 14:16:15.198025 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:15.197776    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb" event={"ID":"a8938794-daba-4785-b9d2-5b8d9870e6a8","Type":"ContainerStarted","Data":"e13b13f079f71ef36d8e17dfa5e89fdceaf47dd714c618b127f660f7164d539e"}
Sep 09 14:16:18.214638 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:18.214603    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb" event={"ID":"a8938794-daba-4785-b9d2-5b8d9870e6a8","Type":"ContainerStarted","Data":"c42316d4d8c5a3bc3d3d3968f63ac51804aaa86187aeb98b9031839909d35134"}
Sep 09 14:16:18.234742 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:18.234694    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dd84d6f45-t56sb" podStartSLOduration=2.348164654 podStartE2EDuration="5.234679259s" totalImagesPullingTime="2.886514605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:16:18.233151769 +0000 UTC m=+311.995947008" watchObservedRunningTime="2026-09-09 14:16:18.234679259 +0000 UTC m=+311.997474500"
Sep 09 14:16:58.633026 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.632995    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fff8bd786-thld2"]
Sep 09 14:16:58.637970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.637950    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.650052 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.650029    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fff8bd786-thld2"]
Sep 09 14:16:58.724171 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.724150    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjpm\" (UniqueName: \"kubernetes.io/projected/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-kube-api-access-ppjpm\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.724280 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.724183    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-service-ca\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.724280 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.724213    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-console-serving-cert\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.724280 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.724230    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-oauth-serving-cert\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.724444 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.724292    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-console-config\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.724444 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.724342    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-trusted-ca-bundle\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.724444 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.724387    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-console-oauth-config\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.824760 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.824739    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjpm\" (UniqueName: \"kubernetes.io/projected/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-kube-api-access-ppjpm\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.824863 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.824773    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-service-ca\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.824863 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.824799    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-console-serving-cert\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.824960 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.824905    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-oauth-serving-cert\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.824960 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.824945    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-console-config\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.825070 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.824974    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-trusted-ca-bundle\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.825070 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.825003    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-console-oauth-config\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.825478 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.825445    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-service-ca\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.825599 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.825575    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-oauth-serving-cert\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.825696 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.825679    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-console-config\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.825759 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.825737    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-trusted-ca-bundle\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.827348 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.827327    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-console-oauth-config\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.827429 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.827394    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-console-serving-cert\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.833254 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.833233    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjpm\" (UniqueName: \"kubernetes.io/projected/63887fb4-3b5d-43ec-ab33-36f8fb16ea03-kube-api-access-ppjpm\") pod \"console-7fff8bd786-thld2\" (UID: \"63887fb4-3b5d-43ec-ab33-36f8fb16ea03\") " pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:58.948323 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:58.948268    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:16:59.218077 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:59.218054    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fff8bd786-thld2"]
Sep 09 14:16:59.220189 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:16:59.220153    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63887fb4_3b5d_43ec_ab33_36f8fb16ea03.slice/crio-682b9cda4753a4dfbb7e33d0d2083be8d0ed98f25c1955ab9f0d45b69b7e0e6d WatchSource:0}: Error finding container 682b9cda4753a4dfbb7e33d0d2083be8d0ed98f25c1955ab9f0d45b69b7e0e6d: Status 404 returned error can't find the container with id 682b9cda4753a4dfbb7e33d0d2083be8d0ed98f25c1955ab9f0d45b69b7e0e6d
Sep 09 14:16:59.407676 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:59.407643    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fff8bd786-thld2" event={"ID":"63887fb4-3b5d-43ec-ab33-36f8fb16ea03","Type":"ContainerStarted","Data":"8d815490428af9e82dfd636e7db30c21e12f743d6171089f8bdbe989b7c7faec"}
Sep 09 14:16:59.407812 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:59.407691    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fff8bd786-thld2" event={"ID":"63887fb4-3b5d-43ec-ab33-36f8fb16ea03","Type":"ContainerStarted","Data":"682b9cda4753a4dfbb7e33d0d2083be8d0ed98f25c1955ab9f0d45b69b7e0e6d"}
Sep 09 14:16:59.426080 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:16:59.426019    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fff8bd786-thld2" podStartSLOduration=1.426004488 podStartE2EDuration="1.426004488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:16:59.425293344 +0000 UTC m=+353.188088584" watchObservedRunningTime="2026-09-09 14:16:59.426004488 +0000 UTC m=+353.188799729"
Sep 09 14:17:08.948922 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:08.948893    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:17:08.949309 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:08.948934    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:17:08.953539 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:08.953518    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:17:09.459594 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:09.459553    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fff8bd786-thld2"
Sep 09 14:17:09.513428 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:09.513388    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d7568d689-njcch"]
Sep 09 14:17:34.534472 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.534419    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d7568d689-njcch" podUID="8b7606f5-a315-467c-b249-5d486fac19cd" containerName="console" containerID="cri-o://5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1" gracePeriod=15
Sep 09 14:17:34.770527 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.770503    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:17:34.825538 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.825482    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-console-config\") pod \"8b7606f5-a315-467c-b249-5d486fac19cd\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") "
Sep 09 14:17:34.825538 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.825522    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-oauth-serving-cert\") pod \"8b7606f5-a315-467c-b249-5d486fac19cd\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") "
Sep 09 14:17:34.825670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.825550    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-serving-cert\") pod \"8b7606f5-a315-467c-b249-5d486fac19cd\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") "
Sep 09 14:17:34.825670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.825587    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-trusted-ca-bundle\") pod \"8b7606f5-a315-467c-b249-5d486fac19cd\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") "
Sep 09 14:17:34.825670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.825602    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-oauth-config\") pod \"8b7606f5-a315-467c-b249-5d486fac19cd\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") "
Sep 09 14:17:34.825670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.825637    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-service-ca\") pod \"8b7606f5-a315-467c-b249-5d486fac19cd\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") "
Sep 09 14:17:34.825823 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.825799    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdnrn\" (UniqueName: \"kubernetes.io/projected/8b7606f5-a315-467c-b249-5d486fac19cd-kube-api-access-fdnrn\") pod \"8b7606f5-a315-467c-b249-5d486fac19cd\" (UID: \"8b7606f5-a315-467c-b249-5d486fac19cd\") "
Sep 09 14:17:34.825963 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.825800    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-console-config" (OuterVolumeSpecName: "console-config") pod "8b7606f5-a315-467c-b249-5d486fac19cd" (UID: "8b7606f5-a315-467c-b249-5d486fac19cd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:17:34.826072 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.826048    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-service-ca" (OuterVolumeSpecName: "service-ca") pod "8b7606f5-a315-467c-b249-5d486fac19cd" (UID: "8b7606f5-a315-467c-b249-5d486fac19cd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:17:34.826072 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.826057    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8b7606f5-a315-467c-b249-5d486fac19cd" (UID: "8b7606f5-a315-467c-b249-5d486fac19cd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:17:34.826208 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.826078    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8b7606f5-a315-467c-b249-5d486fac19cd" (UID: "8b7606f5-a315-467c-b249-5d486fac19cd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:17:34.826259 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.826222    2532 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-oauth-serving-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:17:34.826259 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.826241    2532 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-trusted-ca-bundle\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:17:34.826259 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.826256    2532 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:17:34.826366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.826269    2532 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b7606f5-a315-467c-b249-5d486fac19cd-console-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:17:34.827739 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.827705    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7606f5-a315-467c-b249-5d486fac19cd-kube-api-access-fdnrn" (OuterVolumeSpecName: "kube-api-access-fdnrn") pod "8b7606f5-a315-467c-b249-5d486fac19cd" (UID: "8b7606f5-a315-467c-b249-5d486fac19cd"). InnerVolumeSpecName "kube-api-access-fdnrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:17:34.827739 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.827721    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8b7606f5-a315-467c-b249-5d486fac19cd" (UID: "8b7606f5-a315-467c-b249-5d486fac19cd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:17:34.827872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.827786    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8b7606f5-a315-467c-b249-5d486fac19cd" (UID: "8b7606f5-a315-467c-b249-5d486fac19cd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:17:34.926883 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.926854    2532 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-serving-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:17:34.926883 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.926879    2532 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b7606f5-a315-467c-b249-5d486fac19cd-console-oauth-config\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:17:34.927024 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:34.926893    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdnrn\" (UniqueName: \"kubernetes.io/projected/8b7606f5-a315-467c-b249-5d486fac19cd-kube-api-access-fdnrn\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:17:35.571493 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:35.571462    2532 generic.go:356] "Generic (PLEG): container finished" podID="8b7606f5-a315-467c-b249-5d486fac19cd" containerID="5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1" exitCode=0
Sep 09 14:17:35.571934 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:35.571508    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7568d689-njcch"
Sep 09 14:17:35.571934 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:35.571550    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7568d689-njcch" event={"ID":"8b7606f5-a315-467c-b249-5d486fac19cd","Type":"ContainerDied","Data":"5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1"}
Sep 09 14:17:35.571934 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:35.571607    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7568d689-njcch" event={"ID":"8b7606f5-a315-467c-b249-5d486fac19cd","Type":"ContainerDied","Data":"a2effb5519956ebb5fb631da923b65952824ab0c741b47c4c41c8282a1cbb13a"}
Sep 09 14:17:35.571934 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:35.571639    2532 scope.go:117] "RemoveContainer" containerID="5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1"
Sep 09 14:17:35.583249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:35.583231    2532 scope.go:117] "RemoveContainer" containerID="5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1"
Sep 09 14:17:35.583523 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:17:35.583505    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1\": container with ID starting with 5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1 not found: ID does not exist" containerID="5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1"
Sep 09 14:17:35.583588 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:35.583534    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1"} err="failed to get container status \"5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1\": rpc error: code = NotFound desc = could not find container \"5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1\": container with ID starting with 5f9119499145c076efd918b0cb67985cf32621507ee7efb841fcd9c7d2adf2f1 not found: ID does not exist"
Sep 09 14:17:35.596089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:35.596061    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d7568d689-njcch"]
Sep 09 14:17:35.599329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:35.599311    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d7568d689-njcch"]
Sep 09 14:17:36.733365 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:36.733337    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7606f5-a315-467c-b249-5d486fac19cd" path="/var/lib/kubelet/pods/8b7606f5-a315-467c-b249-5d486fac19cd/volumes"
Sep 09 14:17:58.608087 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.608050    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"]
Sep 09 14:17:58.608476 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.608412    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7606f5-a315-467c-b249-5d486fac19cd" containerName="console"
Sep 09 14:17:58.608476 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.608423    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7606f5-a315-467c-b249-5d486fac19cd" containerName="console"
Sep 09 14:17:58.608563 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.608486    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7606f5-a315-467c-b249-5d486fac19cd" containerName="console"
Sep 09 14:17:58.612737 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.612716    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:58.615185 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.615154    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 14:17:58.615185 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.615173    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 14:17:58.615351 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.615191    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 14:17:58.616118 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.616085    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cf4nw\""
Sep 09 14:17:58.616169 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.616135    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 14:17:58.620988 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.620963    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"]
Sep 09 14:17:58.716661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.716637    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5t5\" (UniqueName: \"kubernetes.io/projected/e027b5d5-e827-4e21-9ccf-0070914e9011-kube-api-access-fs5t5\") pod \"kuadrant-console-plugin-fbdd4f445-pp66l\" (UID: \"e027b5d5-e827-4e21-9ccf-0070914e9011\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:58.716782 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.716739    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e027b5d5-e827-4e21-9ccf-0070914e9011-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pp66l\" (UID: \"e027b5d5-e827-4e21-9ccf-0070914e9011\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:58.716782 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.716760    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e027b5d5-e827-4e21-9ccf-0070914e9011-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pp66l\" (UID: \"e027b5d5-e827-4e21-9ccf-0070914e9011\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:58.818024 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.817995    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5t5\" (UniqueName: \"kubernetes.io/projected/e027b5d5-e827-4e21-9ccf-0070914e9011-kube-api-access-fs5t5\") pod \"kuadrant-console-plugin-fbdd4f445-pp66l\" (UID: \"e027b5d5-e827-4e21-9ccf-0070914e9011\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:58.818223 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.818204    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e027b5d5-e827-4e21-9ccf-0070914e9011-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pp66l\" (UID: \"e027b5d5-e827-4e21-9ccf-0070914e9011\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:58.818311 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.818246    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e027b5d5-e827-4e21-9ccf-0070914e9011-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pp66l\" (UID: \"e027b5d5-e827-4e21-9ccf-0070914e9011\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:58.818392 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:17:58.818371    2532 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 14:17:58.818483 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:17:58.818447    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e027b5d5-e827-4e21-9ccf-0070914e9011-plugin-serving-cert podName:e027b5d5-e827-4e21-9ccf-0070914e9011 nodeName:}" failed. No retries permitted until 2026-09-09 14:17:59.318429563 +0000 UTC m=+413.081224781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/e027b5d5-e827-4e21-9ccf-0070914e9011-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-pp66l" (UID: "e027b5d5-e827-4e21-9ccf-0070914e9011") : secret "plugin-serving-cert" not found
Sep 09 14:17:58.818862 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.818843    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e027b5d5-e827-4e21-9ccf-0070914e9011-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pp66l\" (UID: \"e027b5d5-e827-4e21-9ccf-0070914e9011\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:58.829704 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:58.829676    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5t5\" (UniqueName: \"kubernetes.io/projected/e027b5d5-e827-4e21-9ccf-0070914e9011-kube-api-access-fs5t5\") pod \"kuadrant-console-plugin-fbdd4f445-pp66l\" (UID: \"e027b5d5-e827-4e21-9ccf-0070914e9011\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:59.322460 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:59.322434    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e027b5d5-e827-4e21-9ccf-0070914e9011-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pp66l\" (UID: \"e027b5d5-e827-4e21-9ccf-0070914e9011\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:59.324655 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:59.324633    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e027b5d5-e827-4e21-9ccf-0070914e9011-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pp66l\" (UID: \"e027b5d5-e827-4e21-9ccf-0070914e9011\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:59.522328 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:59.522306    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"
Sep 09 14:17:59.791028 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:17:59.791006    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l"]
Sep 09 14:17:59.793262 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:17:59.793231    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode027b5d5_e827_4e21_9ccf_0070914e9011.slice/crio-0a070e0a1505ab760b01651dae9ee045aa66aa3f1791e9082726f5e73a2b5541 WatchSource:0}: Error finding container 0a070e0a1505ab760b01651dae9ee045aa66aa3f1791e9082726f5e73a2b5541: Status 404 returned error can't find the container with id 0a070e0a1505ab760b01651dae9ee045aa66aa3f1791e9082726f5e73a2b5541
Sep 09 14:18:00.696638 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:00.696594    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l" event={"ID":"e027b5d5-e827-4e21-9ccf-0070914e9011","Type":"ContainerStarted","Data":"0a070e0a1505ab760b01651dae9ee045aa66aa3f1791e9082726f5e73a2b5541"}
Sep 09 14:18:04.721835 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:04.721754    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l" event={"ID":"e027b5d5-e827-4e21-9ccf-0070914e9011","Type":"ContainerStarted","Data":"bd65dfa8c83514d62e4d08ab1d27213b4cf586429efd8f4e86b8f5c040c3cd5e"}
Sep 09 14:18:04.742947 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:04.742893    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pp66l" podStartSLOduration=2.103017011 podStartE2EDuration="6.742877331s" totalImagesPullingTime="4.63986032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:18:04.742078655 +0000 UTC m=+418.504873896" watchObservedRunningTime="2026-09-09 14:18:04.742877331 +0000 UTC m=+418.505672570"
Sep 09 14:18:08.896002 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:08.895966    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"]
Sep 09 14:18:08.899571 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:08.899553    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:08.902031 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:08.902007    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-w9wb4\""
Sep 09 14:18:08.908994 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:08.908835    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/883946e3-5cad-4247-994b-f587b2b57caa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-md2mx\" (UID: \"883946e3-5cad-4247-994b-f587b2b57caa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:08.908994 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:08.908892    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkjm\" (UniqueName: \"kubernetes.io/projected/883946e3-5cad-4247-994b-f587b2b57caa-kube-api-access-hnkjm\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-md2mx\" (UID: \"883946e3-5cad-4247-994b-f587b2b57caa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:08.911033 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:08.911013    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"]
Sep 09 14:18:09.010011 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:09.009984    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/883946e3-5cad-4247-994b-f587b2b57caa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-md2mx\" (UID: \"883946e3-5cad-4247-994b-f587b2b57caa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:09.010137 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:09.010016    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkjm\" (UniqueName: \"kubernetes.io/projected/883946e3-5cad-4247-994b-f587b2b57caa-kube-api-access-hnkjm\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-md2mx\" (UID: \"883946e3-5cad-4247-994b-f587b2b57caa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:09.010330 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:09.010313    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/883946e3-5cad-4247-994b-f587b2b57caa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-md2mx\" (UID: \"883946e3-5cad-4247-994b-f587b2b57caa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:09.027181 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:09.027155    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkjm\" (UniqueName: \"kubernetes.io/projected/883946e3-5cad-4247-994b-f587b2b57caa-kube-api-access-hnkjm\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-md2mx\" (UID: \"883946e3-5cad-4247-994b-f587b2b57caa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:09.210752 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:09.210698    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:09.481712 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:09.481531    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"]
Sep 09 14:18:09.484650 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:18:09.484627    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883946e3_5cad_4247_994b_f587b2b57caa.slice/crio-22ac12e74e74d6d329ac7ff92ae2e8ac765824d449b1e69b308ed5a5e4cd8161 WatchSource:0}: Error finding container 22ac12e74e74d6d329ac7ff92ae2e8ac765824d449b1e69b308ed5a5e4cd8161: Status 404 returned error can't find the container with id 22ac12e74e74d6d329ac7ff92ae2e8ac765824d449b1e69b308ed5a5e4cd8161
Sep 09 14:18:09.751193 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:09.751093    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx" event={"ID":"883946e3-5cad-4247-994b-f587b2b57caa","Type":"ContainerStarted","Data":"22ac12e74e74d6d329ac7ff92ae2e8ac765824d449b1e69b308ed5a5e4cd8161"}
Sep 09 14:18:10.033844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.033766    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"]
Sep 09 14:18:10.048286 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.048252    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"]
Sep 09 14:18:10.082504 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.082449    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"]
Sep 09 14:18:10.085617 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.085591    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:10.100789 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.100332    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"]
Sep 09 14:18:10.118231 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.118195    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz7c4\" (UniqueName: \"kubernetes.io/projected/82547d07-3ab2-4e09-8668-bf3ac2850260-kube-api-access-kz7c4\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-2q2kg\" (UID: \"82547d07-3ab2-4e09-8668-bf3ac2850260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:10.118396 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.118257    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82547d07-3ab2-4e09-8668-bf3ac2850260-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-2q2kg\" (UID: \"82547d07-3ab2-4e09-8668-bf3ac2850260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:10.219044 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.219004    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82547d07-3ab2-4e09-8668-bf3ac2850260-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-2q2kg\" (UID: \"82547d07-3ab2-4e09-8668-bf3ac2850260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:10.219721 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.219439    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7c4\" (UniqueName: \"kubernetes.io/projected/82547d07-3ab2-4e09-8668-bf3ac2850260-kube-api-access-kz7c4\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-2q2kg\" (UID: \"82547d07-3ab2-4e09-8668-bf3ac2850260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:10.220298 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.220274    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82547d07-3ab2-4e09-8668-bf3ac2850260-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-2q2kg\" (UID: \"82547d07-3ab2-4e09-8668-bf3ac2850260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:10.228812 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.228785    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz7c4\" (UniqueName: \"kubernetes.io/projected/82547d07-3ab2-4e09-8668-bf3ac2850260-kube-api-access-kz7c4\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-2q2kg\" (UID: \"82547d07-3ab2-4e09-8668-bf3ac2850260\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:10.400906 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.400761    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:10.694577 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.694544    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"]
Sep 09 14:18:10.697748 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:18:10.697708    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82547d07_3ab2_4e09_8668_bf3ac2850260.slice/crio-23c464e6adda47359c2bc26e66533bd5d2c28aff518954668f4abd951fa98ba3 WatchSource:0}: Error finding container 23c464e6adda47359c2bc26e66533bd5d2c28aff518954668f4abd951fa98ba3: Status 404 returned error can't find the container with id 23c464e6adda47359c2bc26e66533bd5d2c28aff518954668f4abd951fa98ba3
Sep 09 14:18:10.758819 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:10.758781    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg" event={"ID":"82547d07-3ab2-4e09-8668-bf3ac2850260","Type":"ContainerStarted","Data":"23c464e6adda47359c2bc26e66533bd5d2c28aff518954668f4abd951fa98ba3"}
Sep 09 14:18:13.778416 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:13.778341    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg" event={"ID":"82547d07-3ab2-4e09-8668-bf3ac2850260","Type":"ContainerStarted","Data":"281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2"}
Sep 09 14:18:13.778416 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:13.778415    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:13.780092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:13.780065    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx" podUID="883946e3-5cad-4247-994b-f587b2b57caa" containerName="manager" containerID="cri-o://9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f" gracePeriod=2
Sep 09 14:18:13.812478 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:13.812286    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg" podStartSLOduration=0.919140792 podStartE2EDuration="3.812267682s" totalImagesPullingTime="2.89312689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:18:13.81150817 +0000 UTC m=+427.574303423" watchObservedRunningTime="2026-09-09 14:18:13.812267682 +0000 UTC m=+427.575062919"
Sep 09 14:18:14.023385 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.023359    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:14.025662 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:18:14.025635    2532 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-md2mx\" is forbidden: User \"system:node:ip-10-0-139-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-251.ec2.internal' and this object" podUID="883946e3-5cad-4247-994b-f587b2b57caa" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:14.055347 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.055285    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/883946e3-5cad-4247-994b-f587b2b57caa-extensions-socket-volume\") pod \"883946e3-5cad-4247-994b-f587b2b57caa\" (UID: \"883946e3-5cad-4247-994b-f587b2b57caa\") "
Sep 09 14:18:14.055461 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.055345    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnkjm\" (UniqueName: \"kubernetes.io/projected/883946e3-5cad-4247-994b-f587b2b57caa-kube-api-access-hnkjm\") pod \"883946e3-5cad-4247-994b-f587b2b57caa\" (UID: \"883946e3-5cad-4247-994b-f587b2b57caa\") "
Sep 09 14:18:14.055517 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.055494    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883946e3-5cad-4247-994b-f587b2b57caa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "883946e3-5cad-4247-994b-f587b2b57caa" (UID: "883946e3-5cad-4247-994b-f587b2b57caa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:18:14.057426 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.057406    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883946e3-5cad-4247-994b-f587b2b57caa-kube-api-access-hnkjm" (OuterVolumeSpecName: "kube-api-access-hnkjm") pod "883946e3-5cad-4247-994b-f587b2b57caa" (UID: "883946e3-5cad-4247-994b-f587b2b57caa"). InnerVolumeSpecName "kube-api-access-hnkjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:18:14.156778 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.156747    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkjm\" (UniqueName: \"kubernetes.io/projected/883946e3-5cad-4247-994b-f587b2b57caa-kube-api-access-hnkjm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:18:14.156778 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.156776    2532 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/883946e3-5cad-4247-994b-f587b2b57caa-extensions-socket-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:18:14.734022 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.733990    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883946e3-5cad-4247-994b-f587b2b57caa" path="/var/lib/kubelet/pods/883946e3-5cad-4247-994b-f587b2b57caa/volumes"
Sep 09 14:18:14.785726 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.785698    2532 generic.go:356] "Generic (PLEG): container finished" podID="883946e3-5cad-4247-994b-f587b2b57caa" containerID="9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f" exitCode=2
Sep 09 14:18:14.786046 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.785738    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:14.786046 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.785794    2532 scope.go:117] "RemoveContainer" containerID="9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f"
Sep 09 14:18:14.789909 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:18:14.789879    2532 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-md2mx\" is forbidden: User \"system:node:ip-10-0-139-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-251.ec2.internal' and this object" podUID="883946e3-5cad-4247-994b-f587b2b57caa" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:14.793850 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.793833    2532 scope.go:117] "RemoveContainer" containerID="9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f"
Sep 09 14:18:14.794083 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:18:14.794068    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f\": container with ID starting with 9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f not found: ID does not exist" containerID="9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f"
Sep 09 14:18:14.794156 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:14.794090    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f"} err="failed to get container status \"9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f\": rpc error: code = NotFound desc = could not find container \"9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f\": container with ID starting with 9af61c2f4e1de5d48af4a249e5f269cc9bc8d120eb9c2ee87f32a7faf8d1116f not found: ID does not exist"
Sep 09 14:18:16.733463 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:18:16.733424    2532 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-md2mx\" is forbidden: User \"system:node:ip-10-0-139-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-251.ec2.internal' and this object" podUID="883946e3-5cad-4247-994b-f587b2b57caa" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-md2mx"
Sep 09 14:18:24.788021 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:24.787992    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:36.990538 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:36.990455    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"]
Sep 09 14:18:36.990950 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:36.990657    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg" podUID="82547d07-3ab2-4e09-8668-bf3ac2850260" containerName="manager" containerID="cri-o://281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2" gracePeriod=10
Sep 09 14:18:37.775460 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.775435    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:37.850751 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.850723    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82547d07-3ab2-4e09-8668-bf3ac2850260-extensions-socket-volume\") pod \"82547d07-3ab2-4e09-8668-bf3ac2850260\" (UID: \"82547d07-3ab2-4e09-8668-bf3ac2850260\") "
Sep 09 14:18:37.850904 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.850818    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz7c4\" (UniqueName: \"kubernetes.io/projected/82547d07-3ab2-4e09-8668-bf3ac2850260-kube-api-access-kz7c4\") pod \"82547d07-3ab2-4e09-8668-bf3ac2850260\" (UID: \"82547d07-3ab2-4e09-8668-bf3ac2850260\") "
Sep 09 14:18:37.851189 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.851165    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82547d07-3ab2-4e09-8668-bf3ac2850260-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "82547d07-3ab2-4e09-8668-bf3ac2850260" (UID: "82547d07-3ab2-4e09-8668-bf3ac2850260"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:18:37.853125 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.853093    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82547d07-3ab2-4e09-8668-bf3ac2850260-kube-api-access-kz7c4" (OuterVolumeSpecName: "kube-api-access-kz7c4") pod "82547d07-3ab2-4e09-8668-bf3ac2850260" (UID: "82547d07-3ab2-4e09-8668-bf3ac2850260"). InnerVolumeSpecName "kube-api-access-kz7c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:18:37.891300 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.891230    2532 generic.go:356] "Generic (PLEG): container finished" podID="82547d07-3ab2-4e09-8668-bf3ac2850260" containerID="281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2" exitCode=0
Sep 09 14:18:37.891300 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.891270    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"
Sep 09 14:18:37.891441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.891320    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg" event={"ID":"82547d07-3ab2-4e09-8668-bf3ac2850260","Type":"ContainerDied","Data":"281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2"}
Sep 09 14:18:37.891441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.891370    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg" event={"ID":"82547d07-3ab2-4e09-8668-bf3ac2850260","Type":"ContainerDied","Data":"23c464e6adda47359c2bc26e66533bd5d2c28aff518954668f4abd951fa98ba3"}
Sep 09 14:18:37.891441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.891392    2532 scope.go:117] "RemoveContainer" containerID="281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2"
Sep 09 14:18:37.906557 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.906540    2532 scope.go:117] "RemoveContainer" containerID="281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2"
Sep 09 14:18:37.906830 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:18:37.906810    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2\": container with ID starting with 281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2 not found: ID does not exist" containerID="281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2"
Sep 09 14:18:37.906889 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.906840    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2"} err="failed to get container status \"281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2\": rpc error: code = NotFound desc = could not find container \"281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2\": container with ID starting with 281dd42cee99a341ca78f8147de3e7035d5432e1e5e563d3c3852fa85b1263d2 not found: ID does not exist"
Sep 09 14:18:37.915929 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.915905    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"]
Sep 09 14:18:37.922015 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.921989    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-2q2kg"]
Sep 09 14:18:37.951900 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.951876    2532 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82547d07-3ab2-4e09-8668-bf3ac2850260-extensions-socket-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:18:37.951900 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:37.951897    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz7c4\" (UniqueName: \"kubernetes.io/projected/82547d07-3ab2-4e09-8668-bf3ac2850260-kube-api-access-kz7c4\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:18:38.734649 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:38.734614    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82547d07-3ab2-4e09-8668-bf3ac2850260" path="/var/lib/kubelet/pods/82547d07-3ab2-4e09-8668-bf3ac2850260/volumes"
Sep 09 14:18:55.076730 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.076696    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rkrkk"]
Sep 09 14:18:55.077081 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.077033    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82547d07-3ab2-4e09-8668-bf3ac2850260" containerName="manager"
Sep 09 14:18:55.077081 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.077044    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="82547d07-3ab2-4e09-8668-bf3ac2850260" containerName="manager"
Sep 09 14:18:55.077081 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.077069    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883946e3-5cad-4247-994b-f587b2b57caa" containerName="manager"
Sep 09 14:18:55.077081 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.077076    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="883946e3-5cad-4247-994b-f587b2b57caa" containerName="manager"
Sep 09 14:18:55.077252 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.077138    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="883946e3-5cad-4247-994b-f587b2b57caa" containerName="manager"
Sep 09 14:18:55.077252 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.077150    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="82547d07-3ab2-4e09-8668-bf3ac2850260" containerName="manager"
Sep 09 14:18:55.083218 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.083195    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:18:55.085640 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.085570    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 14:18:55.088090 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.088061    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rkrkk"]
Sep 09 14:18:55.172662 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.172632    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rkrkk"]
Sep 09 14:18:55.188082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.188057    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-config-file\") pod \"limitador-limitador-8588749977-rkrkk\" (UID: \"ca6cdaf3-ba52-4078-a4c2-5fe58c851822\") " pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:18:55.188202 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.188134    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gf8\" (UniqueName: \"kubernetes.io/projected/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-kube-api-access-x2gf8\") pod \"limitador-limitador-8588749977-rkrkk\" (UID: \"ca6cdaf3-ba52-4078-a4c2-5fe58c851822\") " pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:18:55.289002 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.288980    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-config-file\") pod \"limitador-limitador-8588749977-rkrkk\" (UID: \"ca6cdaf3-ba52-4078-a4c2-5fe58c851822\") " pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:18:55.289153 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.289027    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gf8\" (UniqueName: \"kubernetes.io/projected/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-kube-api-access-x2gf8\") pod \"limitador-limitador-8588749977-rkrkk\" (UID: \"ca6cdaf3-ba52-4078-a4c2-5fe58c851822\") " pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:18:55.289611 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.289587    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-config-file\") pod \"limitador-limitador-8588749977-rkrkk\" (UID: \"ca6cdaf3-ba52-4078-a4c2-5fe58c851822\") " pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:18:55.310749 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.310727    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gf8\" (UniqueName: \"kubernetes.io/projected/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-kube-api-access-x2gf8\") pod \"limitador-limitador-8588749977-rkrkk\" (UID: \"ca6cdaf3-ba52-4078-a4c2-5fe58c851822\") " pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:18:55.394373 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.394317    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:18:55.667144 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.667118    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rkrkk"]
Sep 09 14:18:55.669069 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:18:55.669046    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6cdaf3_ba52_4078_a4c2_5fe58c851822.slice/crio-5ad1bb2ad690a118c6779bda3927df3d9693025fecdee77d814036b04c6a2a88 WatchSource:0}: Error finding container 5ad1bb2ad690a118c6779bda3927df3d9693025fecdee77d814036b04c6a2a88: Status 404 returned error can't find the container with id 5ad1bb2ad690a118c6779bda3927df3d9693025fecdee77d814036b04c6a2a88
Sep 09 14:18:55.974339 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:55.974266    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk" event={"ID":"ca6cdaf3-ba52-4078-a4c2-5fe58c851822","Type":"ContainerStarted","Data":"5ad1bb2ad690a118c6779bda3927df3d9693025fecdee77d814036b04c6a2a88"}
Sep 09 14:18:56.097463 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.097430    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-85cdq"]
Sep 09 14:18:56.100370 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.100351    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-85cdq"
Sep 09 14:18:56.102978 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.102958    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2qvvt\""
Sep 09 14:18:56.108424 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.108303    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-85cdq"]
Sep 09 14:18:56.197623 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.197596    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px95b\" (UniqueName: \"kubernetes.io/projected/7a04bed6-10bb-42fe-b2d1-b3a0b44afff8-kube-api-access-px95b\") pod \"authorino-59c974bd48-85cdq\" (UID: \"7a04bed6-10bb-42fe-b2d1-b3a0b44afff8\") " pod="kuadrant-system/authorino-59c974bd48-85cdq"
Sep 09 14:18:56.298821 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.298801    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px95b\" (UniqueName: \"kubernetes.io/projected/7a04bed6-10bb-42fe-b2d1-b3a0b44afff8-kube-api-access-px95b\") pod \"authorino-59c974bd48-85cdq\" (UID: \"7a04bed6-10bb-42fe-b2d1-b3a0b44afff8\") " pod="kuadrant-system/authorino-59c974bd48-85cdq"
Sep 09 14:18:56.305941 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.305922    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px95b\" (UniqueName: \"kubernetes.io/projected/7a04bed6-10bb-42fe-b2d1-b3a0b44afff8-kube-api-access-px95b\") pod \"authorino-59c974bd48-85cdq\" (UID: \"7a04bed6-10bb-42fe-b2d1-b3a0b44afff8\") " pod="kuadrant-system/authorino-59c974bd48-85cdq"
Sep 09 14:18:56.410993 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.410974    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-85cdq"
Sep 09 14:18:56.676282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.676254    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-85cdq"]
Sep 09 14:18:56.678983 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:18:56.678955    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a04bed6_10bb_42fe_b2d1_b3a0b44afff8.slice/crio-96e86ae8bf8c39b4317b1f5b64ae86bb4a9e6824d161c5c829c0882a61295371 WatchSource:0}: Error finding container 96e86ae8bf8c39b4317b1f5b64ae86bb4a9e6824d161c5c829c0882a61295371: Status 404 returned error can't find the container with id 96e86ae8bf8c39b4317b1f5b64ae86bb4a9e6824d161c5c829c0882a61295371
Sep 09 14:18:56.979986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.979958    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-85cdq" event={"ID":"7a04bed6-10bb-42fe-b2d1-b3a0b44afff8","Type":"ContainerStarted","Data":"96e86ae8bf8c39b4317b1f5b64ae86bb4a9e6824d161c5c829c0882a61295371"}
Sep 09 14:18:56.981466 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.981438    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk" event={"ID":"ca6cdaf3-ba52-4078-a4c2-5fe58c851822","Type":"ContainerStarted","Data":"0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a"}
Sep 09 14:18:56.981558 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.981497    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:18:56.998803 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:56.998764    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk" podStartSLOduration=0.789080004 podStartE2EDuration="1.998748817s" totalImagesPullingTime="1.209668813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:18:56.997121228 +0000 UTC m=+470.759916470" watchObservedRunningTime="2026-09-09 14:18:56.998748817 +0000 UTC m=+470.761544056"
Sep 09 14:18:58.996497 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:58.996456    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-85cdq" event={"ID":"7a04bed6-10bb-42fe-b2d1-b3a0b44afff8","Type":"ContainerStarted","Data":"41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067"}
Sep 09 14:18:59.013393 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:18:59.013347    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-85cdq" podStartSLOduration=1.158961823 podStartE2EDuration="3.013332596s" totalImagesPullingTime="1.854370773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:18:59.012668591 +0000 UTC m=+472.775463832" watchObservedRunningTime="2026-09-09 14:18:59.013332596 +0000 UTC m=+472.776127835"
Sep 09 14:19:07.990941 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:07.990908    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:19:08.719417 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:08.719386    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rkrkk"]
Sep 09 14:19:08.719669 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:08.719548    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk" podUID="ca6cdaf3-ba52-4078-a4c2-5fe58c851822" containerName="limitador" containerID="cri-o://0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a" gracePeriod=30
Sep 09 14:19:09.253709 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:09.253685    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:19:09.307092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:09.307031    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2gf8\" (UniqueName: \"kubernetes.io/projected/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-kube-api-access-x2gf8\") pod \"ca6cdaf3-ba52-4078-a4c2-5fe58c851822\" (UID: \"ca6cdaf3-ba52-4078-a4c2-5fe58c851822\") "
Sep 09 14:19:09.307237 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:09.307138    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-config-file\") pod \"ca6cdaf3-ba52-4078-a4c2-5fe58c851822\" (UID: \"ca6cdaf3-ba52-4078-a4c2-5fe58c851822\") "
Sep 09 14:19:09.307446 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:09.307425    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-config-file" (OuterVolumeSpecName: "config-file") pod "ca6cdaf3-ba52-4078-a4c2-5fe58c851822" (UID: "ca6cdaf3-ba52-4078-a4c2-5fe58c851822"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:19:09.308949 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:09.308925    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-kube-api-access-x2gf8" (OuterVolumeSpecName: "kube-api-access-x2gf8") pod "ca6cdaf3-ba52-4078-a4c2-5fe58c851822" (UID: "ca6cdaf3-ba52-4078-a4c2-5fe58c851822"). InnerVolumeSpecName "kube-api-access-x2gf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:19:09.408481 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:09.408458    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gf8\" (UniqueName: \"kubernetes.io/projected/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-kube-api-access-x2gf8\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:19:09.408481 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:09.408480    2532 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca6cdaf3-ba52-4078-a4c2-5fe58c851822-config-file\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:19:10.048935 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:10.048901    2532 generic.go:356] "Generic (PLEG): container finished" podID="ca6cdaf3-ba52-4078-a4c2-5fe58c851822" containerID="0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a" exitCode=0
Sep 09 14:19:10.049063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:10.048942    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk"
Sep 09 14:19:10.049063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:10.048988    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk" event={"ID":"ca6cdaf3-ba52-4078-a4c2-5fe58c851822","Type":"ContainerDied","Data":"0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a"}
Sep 09 14:19:10.049063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:10.049034    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rkrkk" event={"ID":"ca6cdaf3-ba52-4078-a4c2-5fe58c851822","Type":"ContainerDied","Data":"5ad1bb2ad690a118c6779bda3927df3d9693025fecdee77d814036b04c6a2a88"}
Sep 09 14:19:10.049063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:10.049058    2532 scope.go:117] "RemoveContainer" containerID="0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a"
Sep 09 14:19:10.058509 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:10.058488    2532 scope.go:117] "RemoveContainer" containerID="0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a"
Sep 09 14:19:10.058783 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:19:10.058765    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a\": container with ID starting with 0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a not found: ID does not exist" containerID="0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a"
Sep 09 14:19:10.058831 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:10.058792    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a"} err="failed to get container status \"0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a\": rpc error: code = NotFound desc = could not find container \"0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a\": container with ID starting with 0a8e9934eb6339bb764407f4d89d4086faa14bc2862e394092f4d3fac0fc414a not found: ID does not exist"
Sep 09 14:19:10.071776 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:10.071751    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rkrkk"]
Sep 09 14:19:10.075603 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:10.075582    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rkrkk"]
Sep 09 14:19:10.733477 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:10.733446    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca6cdaf3-ba52-4078-a4c2-5fe58c851822" path="/var/lib/kubelet/pods/ca6cdaf3-ba52-4078-a4c2-5fe58c851822/volumes"
Sep 09 14:19:11.087517 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.087439    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"]
Sep 09 14:19:11.087964 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.087945    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca6cdaf3-ba52-4078-a4c2-5fe58c851822" containerName="limitador"
Sep 09 14:19:11.088034 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.087968    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca6cdaf3-ba52-4078-a4c2-5fe58c851822" containerName="limitador"
Sep 09 14:19:11.088133 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.088121    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca6cdaf3-ba52-4078-a4c2-5fe58c851822" containerName="limitador"
Sep 09 14:19:11.092280 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.092266    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"
Sep 09 14:19:11.094925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.094903    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-d7gp6\""
Sep 09 14:19:11.095049 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.094906    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 09 14:19:11.095049 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.094968    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 14:19:11.095791 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.095775    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 14:19:11.100880 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.100854    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"]
Sep 09 14:19:11.120869 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.120849    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1ec597b-328b-4a1a-8f47-dcb3bd1440a6-data\") pod \"postgres-69fb4594cf-fv79k\" (UID: \"d1ec597b-328b-4a1a-8f47-dcb3bd1440a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"
Sep 09 14:19:11.120965 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.120907    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88skb\" (UniqueName: \"kubernetes.io/projected/d1ec597b-328b-4a1a-8f47-dcb3bd1440a6-kube-api-access-88skb\") pod \"postgres-69fb4594cf-fv79k\" (UID: \"d1ec597b-328b-4a1a-8f47-dcb3bd1440a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"
Sep 09 14:19:11.221626 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.221602    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88skb\" (UniqueName: \"kubernetes.io/projected/d1ec597b-328b-4a1a-8f47-dcb3bd1440a6-kube-api-access-88skb\") pod \"postgres-69fb4594cf-fv79k\" (UID: \"d1ec597b-328b-4a1a-8f47-dcb3bd1440a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"
Sep 09 14:19:11.221718 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.221664    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1ec597b-328b-4a1a-8f47-dcb3bd1440a6-data\") pod \"postgres-69fb4594cf-fv79k\" (UID: \"d1ec597b-328b-4a1a-8f47-dcb3bd1440a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"
Sep 09 14:19:11.222013 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.221996    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1ec597b-328b-4a1a-8f47-dcb3bd1440a6-data\") pod \"postgres-69fb4594cf-fv79k\" (UID: \"d1ec597b-328b-4a1a-8f47-dcb3bd1440a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"
Sep 09 14:19:11.229930 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.229906    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88skb\" (UniqueName: \"kubernetes.io/projected/d1ec597b-328b-4a1a-8f47-dcb3bd1440a6-kube-api-access-88skb\") pod \"postgres-69fb4594cf-fv79k\" (UID: \"d1ec597b-328b-4a1a-8f47-dcb3bd1440a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"
Sep 09 14:19:11.403837 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.403772    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"
Sep 09 14:19:11.670935 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:11.670851    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"]
Sep 09 14:19:11.673030 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:19:11.673000    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ec597b_328b_4a1a_8f47_dcb3bd1440a6.slice/crio-6995bcd6a5c7f6f7dbf74ed8e495e94c4d83b5bf63e8e20b2baa6a68f34cc974 WatchSource:0}: Error finding container 6995bcd6a5c7f6f7dbf74ed8e495e94c4d83b5bf63e8e20b2baa6a68f34cc974: Status 404 returned error can't find the container with id 6995bcd6a5c7f6f7dbf74ed8e495e94c4d83b5bf63e8e20b2baa6a68f34cc974
Sep 09 14:19:12.060163 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:12.060134    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k" event={"ID":"d1ec597b-328b-4a1a-8f47-dcb3bd1440a6","Type":"ContainerStarted","Data":"6995bcd6a5c7f6f7dbf74ed8e495e94c4d83b5bf63e8e20b2baa6a68f34cc974"}
Sep 09 14:19:14.072408 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:14.072371    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k" event={"ID":"d1ec597b-328b-4a1a-8f47-dcb3bd1440a6","Type":"ContainerStarted","Data":"c5333fb018810b207400745e269d8d77058304a911be204d3855d8a76b2ce4c8"}
Sep 09 14:19:14.072804 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:14.072465    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"
Sep 09 14:19:14.090971 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:14.090929    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k" podStartSLOduration=1.058738203 podStartE2EDuration="3.090916218s" totalImagesPullingTime="2.032178015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:19:14.090234431 +0000 UTC m=+487.853029670" watchObservedRunningTime="2026-09-09 14:19:14.090916218 +0000 UTC m=+487.853711493"
Sep 09 14:19:20.106260 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.106234    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fv79k"
Sep 09 14:19:20.773780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.773750    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-jkmlz"]
Sep 09 14:19:20.779607 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.779589    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jkmlz"
Sep 09 14:19:20.793109 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.793071    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-jkmlz"]
Sep 09 14:19:20.867141 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.867116    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-jkmlz"]
Sep 09 14:19:20.867300 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:19:20.867284    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-d2c6z], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-jkmlz" podUID="c2f3d39c-6302-4a6a-be7d-84dd67b5232f"
Sep 09 14:19:20.893795 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.893770    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4wplp"]
Sep 09 14:19:20.897784 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.897765    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4wplp"
Sep 09 14:19:20.898803 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.898783    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2c6z\" (UniqueName: \"kubernetes.io/projected/c2f3d39c-6302-4a6a-be7d-84dd67b5232f-kube-api-access-d2c6z\") pod \"authorino-557f876f69-jkmlz\" (UID: \"c2f3d39c-6302-4a6a-be7d-84dd67b5232f\") " pod="kuadrant-system/authorino-557f876f69-jkmlz"
Sep 09 14:19:20.900115 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.900081    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 14:19:20.904447 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.904400    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4wplp"]
Sep 09 14:19:20.999163 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.999139    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2c6z\" (UniqueName: \"kubernetes.io/projected/c2f3d39c-6302-4a6a-be7d-84dd67b5232f-kube-api-access-d2c6z\") pod \"authorino-557f876f69-jkmlz\" (UID: \"c2f3d39c-6302-4a6a-be7d-84dd67b5232f\") " pod="kuadrant-system/authorino-557f876f69-jkmlz"
Sep 09 14:19:20.999280 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.999185    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-tls-cert\") pod \"authorino-6d4dfc649f-4wplp\" (UID: \"7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878\") " pod="kuadrant-system/authorino-6d4dfc649f-4wplp"
Sep 09 14:19:20.999280 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:20.999216    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qqg\" (UniqueName: \"kubernetes.io/projected/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-kube-api-access-75qqg\") pod \"authorino-6d4dfc649f-4wplp\" (UID: \"7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878\") " pod="kuadrant-system/authorino-6d4dfc649f-4wplp"
Sep 09 14:19:21.008020 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.007999    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2c6z\" (UniqueName: \"kubernetes.io/projected/c2f3d39c-6302-4a6a-be7d-84dd67b5232f-kube-api-access-d2c6z\") pod \"authorino-557f876f69-jkmlz\" (UID: \"c2f3d39c-6302-4a6a-be7d-84dd67b5232f\") " pod="kuadrant-system/authorino-557f876f69-jkmlz"
Sep 09 14:19:21.014340 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.014307    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4wplp"]
Sep 09 14:19:21.014590 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:19:21.014562    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-75qqg tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-4wplp" podUID="7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878"
Sep 09 14:19:21.042625 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.042572    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65c7c5b795-xkktl"]
Sep 09 14:19:21.046541 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.046525    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c7c5b795-xkktl"
Sep 09 14:19:21.053272 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.053192    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65c7c5b795-xkktl"]
Sep 09 14:19:21.100072 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.100048    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-tls-cert\") pod \"authorino-6d4dfc649f-4wplp\" (UID: \"7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878\") " pod="kuadrant-system/authorino-6d4dfc649f-4wplp"
Sep 09 14:19:21.100178 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.100089    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qqg\" (UniqueName: \"kubernetes.io/projected/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-kube-api-access-75qqg\") pod \"authorino-6d4dfc649f-4wplp\" (UID: \"7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878\") " pod="kuadrant-system/authorino-6d4dfc649f-4wplp"
Sep 09 14:19:21.100178 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.100136    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tsd\" (UniqueName: \"kubernetes.io/projected/6e02c172-899e-4a96-a04b-acff8e17c5ad-kube-api-access-v6tsd\") pod \"authorino-65c7c5b795-xkktl\" (UID: \"6e02c172-899e-4a96-a04b-acff8e17c5ad\") " pod="kuadrant-system/authorino-65c7c5b795-xkktl"
Sep 09 14:19:21.100256 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.100209    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e02c172-899e-4a96-a04b-acff8e17c5ad-tls-cert\") pod \"authorino-65c7c5b795-xkktl\" (UID: \"6e02c172-899e-4a96-a04b-acff8e17c5ad\") " pod="kuadrant-system/authorino-65c7c5b795-xkktl"
Sep 09 14:19:21.102490 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.102461    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-tls-cert\") pod \"authorino-6d4dfc649f-4wplp\" (UID: \"7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878\") " pod="kuadrant-system/authorino-6d4dfc649f-4wplp"
Sep 09 14:19:21.104652 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.104632    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4wplp"
Sep 09 14:19:21.104744 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.104639    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jkmlz"
Sep 09 14:19:21.108253 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.108235    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qqg\" (UniqueName: \"kubernetes.io/projected/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-kube-api-access-75qqg\") pod \"authorino-6d4dfc649f-4wplp\" (UID: \"7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878\") " pod="kuadrant-system/authorino-6d4dfc649f-4wplp"
Sep 09 14:19:21.123577 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.123558    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4wplp"
Sep 09 14:19:21.126931 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.126915    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jkmlz"
Sep 09 14:19:21.200625 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.200603    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75qqg\" (UniqueName: \"kubernetes.io/projected/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-kube-api-access-75qqg\") pod \"7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878\" (UID: \"7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878\") "
Sep 09 14:19:21.200724 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.200644    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-tls-cert\") pod \"7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878\" (UID: \"7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878\") "
Sep 09 14:19:21.200724 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.200684    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2c6z\" (UniqueName: \"kubernetes.io/projected/c2f3d39c-6302-4a6a-be7d-84dd67b5232f-kube-api-access-d2c6z\") pod \"c2f3d39c-6302-4a6a-be7d-84dd67b5232f\" (UID: \"c2f3d39c-6302-4a6a-be7d-84dd67b5232f\") "
Sep 09 14:19:21.200849 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.200799    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tsd\" (UniqueName: \"kubernetes.io/projected/6e02c172-899e-4a96-a04b-acff8e17c5ad-kube-api-access-v6tsd\") pod \"authorino-65c7c5b795-xkktl\" (UID: \"6e02c172-899e-4a96-a04b-acff8e17c5ad\") " pod="kuadrant-system/authorino-65c7c5b795-xkktl"
Sep 09 14:19:21.200906 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.200856    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e02c172-899e-4a96-a04b-acff8e17c5ad-tls-cert\") pod \"authorino-65c7c5b795-xkktl\" (UID: \"6e02c172-899e-4a96-a04b-acff8e17c5ad\") " pod="kuadrant-system/authorino-65c7c5b795-xkktl"
Sep 09 14:19:21.202592 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.202565    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878" (UID: "7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:19:21.202670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.202622    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-kube-api-access-75qqg" (OuterVolumeSpecName: "kube-api-access-75qqg") pod "7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878" (UID: "7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878"). InnerVolumeSpecName "kube-api-access-75qqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:19:21.202724 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.202708    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f3d39c-6302-4a6a-be7d-84dd67b5232f-kube-api-access-d2c6z" (OuterVolumeSpecName: "kube-api-access-d2c6z") pod "c2f3d39c-6302-4a6a-be7d-84dd67b5232f" (UID: "c2f3d39c-6302-4a6a-be7d-84dd67b5232f"). InnerVolumeSpecName "kube-api-access-d2c6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:19:21.203068 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.203054    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e02c172-899e-4a96-a04b-acff8e17c5ad-tls-cert\") pod \"authorino-65c7c5b795-xkktl\" (UID: \"6e02c172-899e-4a96-a04b-acff8e17c5ad\") " pod="kuadrant-system/authorino-65c7c5b795-xkktl"
Sep 09 14:19:21.209600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.209563    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tsd\" (UniqueName: \"kubernetes.io/projected/6e02c172-899e-4a96-a04b-acff8e17c5ad-kube-api-access-v6tsd\") pod \"authorino-65c7c5b795-xkktl\" (UID: \"6e02c172-899e-4a96-a04b-acff8e17c5ad\") " pod="kuadrant-system/authorino-65c7c5b795-xkktl"
Sep 09 14:19:21.301311 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.301250    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qqg\" (UniqueName: \"kubernetes.io/projected/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-kube-api-access-75qqg\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:19:21.301311 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.301274    2532 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878-tls-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:19:21.301311 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.301287    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2c6z\" (UniqueName: \"kubernetes.io/projected/c2f3d39c-6302-4a6a-be7d-84dd67b5232f-kube-api-access-d2c6z\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:19:21.356188 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.356168    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c7c5b795-xkktl"
Sep 09 14:19:21.622695 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:21.622667    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65c7c5b795-xkktl"]
Sep 09 14:19:21.625125 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:19:21.625084    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e02c172_899e_4a96_a04b_acff8e17c5ad.slice/crio-35f0aca8a71ae0b1525cfb90841f97dac291ad9427767235a9e24644a5a59ad7 WatchSource:0}: Error finding container 35f0aca8a71ae0b1525cfb90841f97dac291ad9427767235a9e24644a5a59ad7: Status 404 returned error can't find the container with id 35f0aca8a71ae0b1525cfb90841f97dac291ad9427767235a9e24644a5a59ad7
Sep 09 14:19:22.112529 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.112472    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jkmlz"
Sep 09 14:19:22.112937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.112465    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c7c5b795-xkktl" event={"ID":"6e02c172-899e-4a96-a04b-acff8e17c5ad","Type":"ContainerStarted","Data":"9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12"}
Sep 09 14:19:22.112937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.112612    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c7c5b795-xkktl" event={"ID":"6e02c172-899e-4a96-a04b-acff8e17c5ad","Type":"ContainerStarted","Data":"35f0aca8a71ae0b1525cfb90841f97dac291ad9427767235a9e24644a5a59ad7"}
Sep 09 14:19:22.112937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.112661    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4wplp"
Sep 09 14:19:22.131399 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.131355    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65c7c5b795-xkktl" podStartSLOduration=0.775951204 podStartE2EDuration="1.131342041s" totalImagesPullingTime="355.390837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:19:22.129450975 +0000 UTC m=+495.892246216" watchObservedRunningTime="2026-09-09 14:19:22.131342041 +0000 UTC m=+495.894137293"
Sep 09 14:19:22.155729 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.155698    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-85cdq"]
Sep 09 14:19:22.155925 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.155900    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-85cdq" podUID="7a04bed6-10bb-42fe-b2d1-b3a0b44afff8" containerName="authorino" containerID="cri-o://41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067" gracePeriod=30
Sep 09 14:19:22.161741 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.161707    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4wplp"]
Sep 09 14:19:22.168113 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.168071    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4wplp"]
Sep 09 14:19:22.188599 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.188576    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-jkmlz"]
Sep 09 14:19:22.194487 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.194464    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-jkmlz"]
Sep 09 14:19:22.400536 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.400507    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-85cdq"
Sep 09 14:19:22.511482 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.511458    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px95b\" (UniqueName: \"kubernetes.io/projected/7a04bed6-10bb-42fe-b2d1-b3a0b44afff8-kube-api-access-px95b\") pod \"7a04bed6-10bb-42fe-b2d1-b3a0b44afff8\" (UID: \"7a04bed6-10bb-42fe-b2d1-b3a0b44afff8\") "
Sep 09 14:19:22.513374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.513353    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a04bed6-10bb-42fe-b2d1-b3a0b44afff8-kube-api-access-px95b" (OuterVolumeSpecName: "kube-api-access-px95b") pod "7a04bed6-10bb-42fe-b2d1-b3a0b44afff8" (UID: "7a04bed6-10bb-42fe-b2d1-b3a0b44afff8"). InnerVolumeSpecName "kube-api-access-px95b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:19:22.612790 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.612737    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px95b\" (UniqueName: \"kubernetes.io/projected/7a04bed6-10bb-42fe-b2d1-b3a0b44afff8-kube-api-access-px95b\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:19:22.733676 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.733650    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878" path="/var/lib/kubelet/pods/7e2dcbab-9ee9-4b55-82a3-c1b3c1c5b878/volumes"
Sep 09 14:19:22.733870 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:22.733858    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2f3d39c-6302-4a6a-be7d-84dd67b5232f" path="/var/lib/kubelet/pods/c2f3d39c-6302-4a6a-be7d-84dd67b5232f/volumes"
Sep 09 14:19:23.118205 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:23.118172    2532 generic.go:356] "Generic (PLEG): container finished" podID="7a04bed6-10bb-42fe-b2d1-b3a0b44afff8" containerID="41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067" exitCode=0
Sep 09 14:19:23.118662 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:23.118209    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-85cdq"
Sep 09 14:19:23.118662 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:23.118265    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-85cdq" event={"ID":"7a04bed6-10bb-42fe-b2d1-b3a0b44afff8","Type":"ContainerDied","Data":"41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067"}
Sep 09 14:19:23.118662 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:23.118318    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-85cdq" event={"ID":"7a04bed6-10bb-42fe-b2d1-b3a0b44afff8","Type":"ContainerDied","Data":"96e86ae8bf8c39b4317b1f5b64ae86bb4a9e6824d161c5c829c0882a61295371"}
Sep 09 14:19:23.118662 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:23.118344    2532 scope.go:117] "RemoveContainer" containerID="41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067"
Sep 09 14:19:23.126644 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:23.126627    2532 scope.go:117] "RemoveContainer" containerID="41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067"
Sep 09 14:19:23.126916 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:19:23.126898    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067\": container with ID starting with 41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067 not found: ID does not exist" containerID="41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067"
Sep 09 14:19:23.126965 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:23.126923    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067"} err="failed to get container status \"41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067\": rpc error: code = NotFound desc = could not find container \"41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067\": container with ID starting with 41ca9aaf60616305441552277be6acc3983e779da630db1e8514671bd924c067 not found: ID does not exist"
Sep 09 14:19:23.133921 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:23.133899    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-85cdq"]
Sep 09 14:19:23.136569 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:23.136551    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-85cdq"]
Sep 09 14:19:24.733648 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:19:24.733616    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a04bed6-10bb-42fe-b2d1-b3a0b44afff8" path="/var/lib/kubelet/pods/7a04bed6-10bb-42fe-b2d1-b3a0b44afff8/volumes"
Sep 09 14:20:04.821621 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.821587    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"]
Sep 09 14:20:04.822052 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.822000    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a04bed6-10bb-42fe-b2d1-b3a0b44afff8" containerName="authorino"
Sep 09 14:20:04.822052 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.822013    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a04bed6-10bb-42fe-b2d1-b3a0b44afff8" containerName="authorino"
Sep 09 14:20:04.822146 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.822085    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a04bed6-10bb-42fe-b2d1-b3a0b44afff8" containerName="authorino"
Sep 09 14:20:04.825254 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.825238    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.828602 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.828581    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 14:20:04.828602 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.828590    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 14:20:04.828602 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.828581    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fkrmg\""
Sep 09 14:20:04.835643 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.835601    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"]
Sep 09 14:20:04.866903 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.866875    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-metrics-certs\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.867046 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.866911    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaa9dee-1da4-4a40-a010-56391d76380b-openshift-service-ca\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.867046 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.866963    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-maas-api-tls\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.867149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.867049    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pzj5\" (UniqueName: \"kubernetes.io/projected/dfaa9dee-1da4-4a40-a010-56391d76380b-kube-api-access-9pzj5\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.967638 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.967612    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-metrics-certs\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.967823 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.967644    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaa9dee-1da4-4a40-a010-56391d76380b-openshift-service-ca\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.967823 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.967666    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-maas-api-tls\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.967823 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.967723    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pzj5\" (UniqueName: \"kubernetes.io/projected/dfaa9dee-1da4-4a40-a010-56391d76380b-kube-api-access-9pzj5\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.968391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.968366    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaa9dee-1da4-4a40-a010-56391d76380b-openshift-service-ca\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.970127 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.970083    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-maas-api-tls\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.970230 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.970147    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-metrics-certs\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:04.976373 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:04.976354    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pzj5\" (UniqueName: \"kubernetes.io/projected/dfaa9dee-1da4-4a40-a010-56391d76380b-kube-api-access-9pzj5\") pod \"maas-api-85cbb79749-n8lpn\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") " pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:05.136748 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.136671    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:05.386899 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.386815    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"]
Sep 09 14:20:05.390589 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.390567    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"
Sep 09 14:20:05.393076 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.393052    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7tp9l\""
Sep 09 14:20:05.393217 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.393053    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 14:20:05.402846 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.402823    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"]
Sep 09 14:20:05.415744 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.415725    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"]
Sep 09 14:20:05.472277 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.472250    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2301e4ac-b43b-44f5-8b90-4dffed7fa817-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-m9gkd\" (UID: \"2301e4ac-b43b-44f5-8b90-4dffed7fa817\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"
Sep 09 14:20:05.472431 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.472305    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fssc\" (UniqueName: \"kubernetes.io/projected/2301e4ac-b43b-44f5-8b90-4dffed7fa817-kube-api-access-5fssc\") pod \"payload-pre-processing-59fbc47bf8-m9gkd\" (UID: \"2301e4ac-b43b-44f5-8b90-4dffed7fa817\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"
Sep 09 14:20:05.573537 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.573501    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2301e4ac-b43b-44f5-8b90-4dffed7fa817-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-m9gkd\" (UID: \"2301e4ac-b43b-44f5-8b90-4dffed7fa817\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"
Sep 09 14:20:05.573717 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.573556    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fssc\" (UniqueName: \"kubernetes.io/projected/2301e4ac-b43b-44f5-8b90-4dffed7fa817-kube-api-access-5fssc\") pod \"payload-pre-processing-59fbc47bf8-m9gkd\" (UID: \"2301e4ac-b43b-44f5-8b90-4dffed7fa817\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"
Sep 09 14:20:05.574298 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.574271    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2301e4ac-b43b-44f5-8b90-4dffed7fa817-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-m9gkd\" (UID: \"2301e4ac-b43b-44f5-8b90-4dffed7fa817\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"
Sep 09 14:20:05.581580 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.581556    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fssc\" (UniqueName: \"kubernetes.io/projected/2301e4ac-b43b-44f5-8b90-4dffed7fa817-kube-api-access-5fssc\") pod \"payload-pre-processing-59fbc47bf8-m9gkd\" (UID: \"2301e4ac-b43b-44f5-8b90-4dffed7fa817\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"
Sep 09 14:20:05.704019 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.703923    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"
Sep 09 14:20:05.977757 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:05.977728    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"]
Sep 09 14:20:05.979875 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:20:05.979842    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2301e4ac_b43b_44f5_8b90_4dffed7fa817.slice/crio-a4c8f349ba9d4d9ddc896bd8e03a1a0574ddabb04acf770bc4de209e99fb97a6 WatchSource:0}: Error finding container a4c8f349ba9d4d9ddc896bd8e03a1a0574ddabb04acf770bc4de209e99fb97a6: Status 404 returned error can't find the container with id a4c8f349ba9d4d9ddc896bd8e03a1a0574ddabb04acf770bc4de209e99fb97a6
Sep 09 14:20:06.322690 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:06.322649    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd" event={"ID":"2301e4ac-b43b-44f5-8b90-4dffed7fa817","Type":"ContainerStarted","Data":"a4c8f349ba9d4d9ddc896bd8e03a1a0574ddabb04acf770bc4de209e99fb97a6"}
Sep 09 14:20:06.324429 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:06.324400    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn" event={"ID":"dfaa9dee-1da4-4a40-a010-56391d76380b","Type":"ContainerStarted","Data":"e6e6a705ed5da3618d14d9c145687147d66fafe297221762b0537317459752de"}
Sep 09 14:20:09.342404 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:09.342366    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd" event={"ID":"2301e4ac-b43b-44f5-8b90-4dffed7fa817","Type":"ContainerStarted","Data":"583628be7c8c829d19fc69a390bb027f2b15def485d54ac1f8b572b9ea5cd4fc"}
Sep 09 14:20:09.342853 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:09.342502    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"
Sep 09 14:20:09.344234 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:09.344210    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn" event={"ID":"dfaa9dee-1da4-4a40-a010-56391d76380b","Type":"ContainerStarted","Data":"64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e"}
Sep 09 14:20:09.344358 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:09.344264    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:09.359602 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:09.359563    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd" podStartSLOduration=1.862021851 podStartE2EDuration="4.359551252s" totalImagesPullingTime="2.497529401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:20:09.356979041 +0000 UTC m=+543.119774305" watchObservedRunningTime="2026-09-09 14:20:09.359551252 +0000 UTC m=+543.122346483"
Sep 09 14:20:09.374887 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:09.374844    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn" podStartSLOduration=2.321397309 podStartE2EDuration="5.374832603s" totalImagesPullingTime="3.053435294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:20:09.372455665 +0000 UTC m=+543.135250904" watchObservedRunningTime="2026-09-09 14:20:09.374832603 +0000 UTC m=+543.137627842"
Sep 09 14:20:15.354459 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:15.354429    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:20:20.351079 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:20.351053    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-m9gkd"
Sep 09 14:20:50.494683 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.494651    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"]
Sep 09 14:20:50.499951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.499931    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.502605 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.502580    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-scw2n\""
Sep 09 14:20:50.502726 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.502682    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 14:20:50.503305 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.503288    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 14:20:50.503372 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.503306    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 14:20:50.508365 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.508341    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"]
Sep 09 14:20:50.652790 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.652761    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4af6929-882f-4325-ab65-022e8ae15afe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.652917 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.652824    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.652917 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.652869    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.652917 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.652894    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjzm\" (UniqueName: \"kubernetes.io/projected/c4af6929-882f-4325-ab65-022e8ae15afe-kube-api-access-pqjzm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.653026 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.652945    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.653026 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.652980    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.754121 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.754040    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.754121 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.754081    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4af6929-882f-4325-ab65-022e8ae15afe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.754294 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.754151    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.754294 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.754166    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.754294 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.754183    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjzm\" (UniqueName: \"kubernetes.io/projected/c4af6929-882f-4325-ab65-022e8ae15afe-kube-api-access-pqjzm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.754294 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.754224    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.754517 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.754465    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.754635 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.754522    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.754635 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.754574    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.756406 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.756384    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4af6929-882f-4325-ab65-022e8ae15afe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.756562 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.756544    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4af6929-882f-4325-ab65-022e8ae15afe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.762593 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.762573    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjzm\" (UniqueName: \"kubernetes.io/projected/c4af6929-882f-4325-ab65-022e8ae15afe-kube-api-access-pqjzm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2b2ms\" (UID: \"c4af6929-882f-4325-ab65-022e8ae15afe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:50.811481 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:50.811460    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:20:51.082528 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:51.082458    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"]
Sep 09 14:20:51.085775 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:20:51.085747    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4af6929_882f_4325_ab65_022e8ae15afe.slice/crio-3807eb05e575402484c3f8a413b4e9d19b0f9b8000593db1d845afb553d13f6e WatchSource:0}: Error finding container 3807eb05e575402484c3f8a413b4e9d19b0f9b8000593db1d845afb553d13f6e: Status 404 returned error can't find the container with id 3807eb05e575402484c3f8a413b4e9d19b0f9b8000593db1d845afb553d13f6e
Sep 09 14:20:51.545829 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:51.545789    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms" event={"ID":"c4af6929-882f-4325-ab65-022e8ae15afe","Type":"ContainerStarted","Data":"3807eb05e575402484c3f8a413b4e9d19b0f9b8000593db1d845afb553d13f6e"}
Sep 09 14:20:52.552386 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:52.552347    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms" event={"ID":"c4af6929-882f-4325-ab65-022e8ae15afe","Type":"ContainerStarted","Data":"2ba647baeba1dcfd31adc8e290d5dac49fccb7de8d3138d7e062eeabda89f5b8"}
Sep 09 14:20:52.573576 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:20:52.573532    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms" podStartSLOduration=1.482500118 podStartE2EDuration="2.573516302s" totalImagesPullingTime="1.091016184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:20:52.571197389 +0000 UTC m=+586.333992632" watchObservedRunningTime="2026-09-09 14:20:52.573516302 +0000 UTC m=+586.336311541"
Sep 09 14:21:00.812486 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:00.812453    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:21:00.812880 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:00.812510    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:21:00.824909 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:00.824884    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:21:01.612200 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:01.612171    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2b2ms"
Sep 09 14:21:06.659701 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:06.659671    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:21:06.660161 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:06.659843    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:21:18.707242 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.707210    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"]
Sep 09 14:21:18.710768 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.710752    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.713249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.713226    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 14:21:18.721358 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.721335    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"]
Sep 09 14:21:18.791954 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.791921    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxld\" (UniqueName: \"kubernetes.io/projected/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-kube-api-access-rdxld\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.792134 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.791994    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.792134 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.792056    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.792255 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.792179    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.792255 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.792245    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.792484 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.792422    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.892948 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.892918    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxld\" (UniqueName: \"kubernetes.io/projected/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-kube-api-access-rdxld\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.893068 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.892954    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.893068 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.892982    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.893068 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.893029    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.893068 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.893060    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.893317 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.893090    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.893428 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.893409    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.893464 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.893440    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.893464 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.893450    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.895233 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.895217    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.895396 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.895380    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:18.900412 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:18.900389    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxld\" (UniqueName: \"kubernetes.io/projected/c52f20f6-a72a-435f-9a8f-6109ac8bd26a-kube-api-access-rdxld\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz\" (UID: \"c52f20f6-a72a-435f-9a8f-6109ac8bd26a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:19.021559 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:19.021538    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:19.290878 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:19.290720    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"]
Sep 09 14:21:19.293128 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:21:19.293088    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52f20f6_a72a_435f_9a8f_6109ac8bd26a.slice/crio-5e88387e1559ecdd78774c086a488a031377b1b97e51ce218dcb8b115c9c649d WatchSource:0}: Error finding container 5e88387e1559ecdd78774c086a488a031377b1b97e51ce218dcb8b115c9c649d: Status 404 returned error can't find the container with id 5e88387e1559ecdd78774c086a488a031377b1b97e51ce218dcb8b115c9c649d
Sep 09 14:21:19.294562 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:19.294541    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 14:21:19.685348 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:19.685307    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz" event={"ID":"c52f20f6-a72a-435f-9a8f-6109ac8bd26a","Type":"ContainerStarted","Data":"3bcd0e20a8b8278297f875fc89628957840fdb2850a5f0e56589961c7997f1b0"}
Sep 09 14:21:19.685484 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:19.685406    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz" event={"ID":"c52f20f6-a72a-435f-9a8f-6109ac8bd26a","Type":"ContainerStarted","Data":"5e88387e1559ecdd78774c086a488a031377b1b97e51ce218dcb8b115c9c649d"}
Sep 09 14:21:19.704086 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:19.704043    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz" podStartSLOduration=1.46241692 podStartE2EDuration="1.704025985s" totalImagesPullingTime="241.609065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:21:19.702764445 +0000 UTC m=+613.465559677" watchObservedRunningTime="2026-09-09 14:21:19.704025985 +0000 UTC m=+613.466821226"
Sep 09 14:21:29.022293 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:29.022213    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:29.022293 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:29.022264    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:29.034443 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:29.034419    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:29.743519 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:29.743485    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz"
Sep 09 14:21:38.987310 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:38.987277    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"]
Sep 09 14:21:38.990207 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:38.990185    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:38.992723 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:38.992699    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 14:21:39.001826 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.001803    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"]
Sep 09 14:21:39.170843 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.170809    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.170986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.170871    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv5ln\" (UniqueName: \"kubernetes.io/projected/57935d16-670b-4427-b455-449aded29fc1-kube-api-access-wv5ln\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.170986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.170950    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57935d16-670b-4427-b455-449aded29fc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.171075 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.170998    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.171141 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.171064    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.171141 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.171121    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.272122 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.272038    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv5ln\" (UniqueName: \"kubernetes.io/projected/57935d16-670b-4427-b455-449aded29fc1-kube-api-access-wv5ln\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.272122 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.272078    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57935d16-670b-4427-b455-449aded29fc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.272318 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.272126    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.272318 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.272295    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.272429 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.272347    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.272429 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.272396    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.272538 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.272490    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.272621 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.272604    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.272662 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.272631    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.274427 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.274401    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57935d16-670b-4427-b455-449aded29fc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.274553 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.274536    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57935d16-670b-4427-b455-449aded29fc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.279474 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.279447    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv5ln\" (UniqueName: \"kubernetes.io/projected/57935d16-670b-4427-b455-449aded29fc1-kube-api-access-wv5ln\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg\" (UID: \"57935d16-670b-4427-b455-449aded29fc1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.301253 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.301231    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:39.572260 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.572227    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"]
Sep 09 14:21:39.577937 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:21:39.577903    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57935d16_670b_4427_b455_449aded29fc1.slice/crio-fea16acc6f04a42fd2e14a5fbd6a5fa86f9343d0ff0d28354d481e365f86cd79 WatchSource:0}: Error finding container fea16acc6f04a42fd2e14a5fbd6a5fa86f9343d0ff0d28354d481e365f86cd79: Status 404 returned error can't find the container with id fea16acc6f04a42fd2e14a5fbd6a5fa86f9343d0ff0d28354d481e365f86cd79
Sep 09 14:21:39.777891 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:39.777859    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg" event={"ID":"57935d16-670b-4427-b455-449aded29fc1","Type":"ContainerStarted","Data":"fea16acc6f04a42fd2e14a5fbd6a5fa86f9343d0ff0d28354d481e365f86cd79"}
Sep 09 14:21:40.783877 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:40.783839    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg" event={"ID":"57935d16-670b-4427-b455-449aded29fc1","Type":"ContainerStarted","Data":"4d2de2e4cddca68bce26445b4605101bd98abb6dce2b9f30c9bb5711abaffee5"}
Sep 09 14:21:40.803258 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:40.803208    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg" podStartSLOduration=2.582075626 podStartE2EDuration="2.803192509s" totalImagesPullingTime="221.116883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:21:40.801424983 +0000 UTC m=+634.564220216" watchObservedRunningTime="2026-09-09 14:21:40.803192509 +0000 UTC m=+634.565987748"
Sep 09 14:21:49.302281 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:49.302243    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:49.302653 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:49.302303    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:49.314623 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:49.314601    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:21:49.835714 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:21:49.835684    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg"
Sep 09 14:23:33.537962 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:33.537929    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65c7c5b795-xkktl"]
Sep 09 14:23:33.538599 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:33.538152    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-65c7c5b795-xkktl" podUID="6e02c172-899e-4a96-a04b-acff8e17c5ad" containerName="authorino" containerID="cri-o://9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12" gracePeriod=30
Sep 09 14:23:33.773121 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:33.773080    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c7c5b795-xkktl"
Sep 09 14:23:33.845441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:33.845377    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e02c172-899e-4a96-a04b-acff8e17c5ad-tls-cert\") pod \"6e02c172-899e-4a96-a04b-acff8e17c5ad\" (UID: \"6e02c172-899e-4a96-a04b-acff8e17c5ad\") "
Sep 09 14:23:33.845441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:33.845412    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6tsd\" (UniqueName: \"kubernetes.io/projected/6e02c172-899e-4a96-a04b-acff8e17c5ad-kube-api-access-v6tsd\") pod \"6e02c172-899e-4a96-a04b-acff8e17c5ad\" (UID: \"6e02c172-899e-4a96-a04b-acff8e17c5ad\") "
Sep 09 14:23:33.847421 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:33.847396    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e02c172-899e-4a96-a04b-acff8e17c5ad-kube-api-access-v6tsd" (OuterVolumeSpecName: "kube-api-access-v6tsd") pod "6e02c172-899e-4a96-a04b-acff8e17c5ad" (UID: "6e02c172-899e-4a96-a04b-acff8e17c5ad"). InnerVolumeSpecName "kube-api-access-v6tsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:23:33.855489 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:33.855465    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e02c172-899e-4a96-a04b-acff8e17c5ad-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6e02c172-899e-4a96-a04b-acff8e17c5ad" (UID: "6e02c172-899e-4a96-a04b-acff8e17c5ad"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:23:33.946974 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:33.946950    2532 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e02c172-899e-4a96-a04b-acff8e17c5ad-tls-cert\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:23:33.946974 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:33.946973    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6tsd\" (UniqueName: \"kubernetes.io/projected/6e02c172-899e-4a96-a04b-acff8e17c5ad-kube-api-access-v6tsd\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:23:34.323673 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:34.323640    2532 generic.go:356] "Generic (PLEG): container finished" podID="6e02c172-899e-4a96-a04b-acff8e17c5ad" containerID="9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12" exitCode=0
Sep 09 14:23:34.323837 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:34.323687    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c7c5b795-xkktl"
Sep 09 14:23:34.323837 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:34.323698    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c7c5b795-xkktl" event={"ID":"6e02c172-899e-4a96-a04b-acff8e17c5ad","Type":"ContainerDied","Data":"9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12"}
Sep 09 14:23:34.323837 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:34.323728    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c7c5b795-xkktl" event={"ID":"6e02c172-899e-4a96-a04b-acff8e17c5ad","Type":"ContainerDied","Data":"35f0aca8a71ae0b1525cfb90841f97dac291ad9427767235a9e24644a5a59ad7"}
Sep 09 14:23:34.323837 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:34.323752    2532 scope.go:117] "RemoveContainer" containerID="9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12"
Sep 09 14:23:34.332553 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:34.332524    2532 scope.go:117] "RemoveContainer" containerID="9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12"
Sep 09 14:23:34.332821 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:23:34.332802    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12\": container with ID starting with 9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12 not found: ID does not exist" containerID="9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12"
Sep 09 14:23:34.332883 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:34.332834    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12"} err="failed to get container status \"9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12\": rpc error: code = NotFound desc = could not find container \"9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12\": container with ID starting with 9e607d63dfdf0418583ee770e05522333a5f3862d88a59c863ef95c1f179eb12 not found: ID does not exist"
Sep 09 14:23:34.345720 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:34.345695    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65c7c5b795-xkktl"]
Sep 09 14:23:34.350492 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:34.350473    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65c7c5b795-xkktl"]
Sep 09 14:23:34.734201 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:34.734168    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e02c172-899e-4a96-a04b-acff8e17c5ad" path="/var/lib/kubelet/pods/6e02c172-899e-4a96-a04b-acff8e17c5ad/volumes"
Sep 09 14:23:56.885501 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:56.885464    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"]
Sep 09 14:23:56.886081 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:56.886061    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e02c172-899e-4a96-a04b-acff8e17c5ad" containerName="authorino"
Sep 09 14:23:56.886172 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:56.886085    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e02c172-899e-4a96-a04b-acff8e17c5ad" containerName="authorino"
Sep 09 14:23:56.886268 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:56.886253    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e02c172-899e-4a96-a04b-acff8e17c5ad" containerName="authorino"
Sep 09 14:23:56.890820 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:56.890799    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:56.893649 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:56.893626    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a0fa5a-serving-cert\""
Sep 09 14:23:56.901717 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:56.901694    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"]
Sep 09 14:23:57.040348 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.040317    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-maas-api-tls\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.040492 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.040352    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.040492 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.040411    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvrd8\" (UniqueName: \"kubernetes.io/projected/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-kube-api-access-hvrd8\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.040492 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.040448    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-metrics-certs\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.141645 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.141571    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-maas-api-tls\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.141645 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.141609    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.141826 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.141674    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvrd8\" (UniqueName: \"kubernetes.io/projected/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-kube-api-access-hvrd8\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.141826 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:23:57.141708    2532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-serving-cert: secret "maas-api-e2e-inf-a-a0fa5a-serving-cert" not found
Sep 09 14:23:57.141826 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:23:57.141776    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-maas-api-tls podName:6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9 nodeName:}" failed. No retries permitted until 2026-09-09 14:23:57.64176086 +0000 UTC m=+771.404556083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-maas-api-tls") pod "maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf" (UID: "6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9") : secret "maas-api-e2e-inf-a-a0fa5a-serving-cert" not found
Sep 09 14:23:57.141826 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.141714    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-metrics-certs\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.142256 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.142237    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.144128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.144088    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-metrics-certs\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.154909 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.154881    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvrd8\" (UniqueName: \"kubernetes.io/projected/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-kube-api-access-hvrd8\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.647435 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.647404    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-maas-api-tls\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.649639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.649621    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-maas-api-tls\") pod \"maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.680444 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.680415    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"]
Sep 09 14:23:57.684253 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.684236    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:23:57.686902 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.686880    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a0fa5a\""
Sep 09 14:23:57.696664 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.696636    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"]
Sep 09 14:23:57.802924 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.802893    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:23:57.849494 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.849451    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n24nl\" (UniqueName: \"kubernetes.io/projected/95a390b7-6abc-440b-9513-5e06d611abd7-kube-api-access-n24nl\") pod \"payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r\" (UID: \"95a390b7-6abc-440b-9513-5e06d611abd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:23:57.849663 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.849520    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a390b7-6abc-440b-9513-5e06d611abd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r\" (UID: \"95a390b7-6abc-440b-9513-5e06d611abd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:23:57.950578 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.950503    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a390b7-6abc-440b-9513-5e06d611abd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r\" (UID: \"95a390b7-6abc-440b-9513-5e06d611abd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:23:57.950918 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.950609    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n24nl\" (UniqueName: \"kubernetes.io/projected/95a390b7-6abc-440b-9513-5e06d611abd7-kube-api-access-n24nl\") pod \"payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r\" (UID: \"95a390b7-6abc-440b-9513-5e06d611abd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:23:57.951176 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.951156    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a390b7-6abc-440b-9513-5e06d611abd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r\" (UID: \"95a390b7-6abc-440b-9513-5e06d611abd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:23:57.958598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.958577    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n24nl\" (UniqueName: \"kubernetes.io/projected/95a390b7-6abc-440b-9513-5e06d611abd7-kube-api-access-n24nl\") pod \"payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r\" (UID: \"95a390b7-6abc-440b-9513-5e06d611abd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:23:57.996135 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:57.996115    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:23:58.082199 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:58.082171    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"]
Sep 09 14:23:58.084053 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:23:58.084026    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6c4f81_a100_4d0f_bdcb_6144b27d3ab9.slice/crio-4d5f39d5482fcb86a381a1f6e0725c10e6e8a0420aa578f94f2424edefa3fc1a WatchSource:0}: Error finding container 4d5f39d5482fcb86a381a1f6e0725c10e6e8a0420aa578f94f2424edefa3fc1a: Status 404 returned error can't find the container with id 4d5f39d5482fcb86a381a1f6e0725c10e6e8a0420aa578f94f2424edefa3fc1a
Sep 09 14:23:58.267967 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:58.267943    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"]
Sep 09 14:23:58.269462 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:23:58.269437    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a390b7_6abc_440b_9513_5e06d611abd7.slice/crio-e171796704ff7a25fa106ce9246c005916d6173e7d7227266102595a4dd8dce5 WatchSource:0}: Error finding container e171796704ff7a25fa106ce9246c005916d6173e7d7227266102595a4dd8dce5: Status 404 returned error can't find the container with id e171796704ff7a25fa106ce9246c005916d6173e7d7227266102595a4dd8dce5
Sep 09 14:23:58.441389 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:58.441358    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r" event={"ID":"95a390b7-6abc-440b-9513-5e06d611abd7","Type":"ContainerStarted","Data":"510f36dcd65aa665998e5120b50005bfb0d22b12d89af2537fc369a4c122e54c"}
Sep 09 14:23:58.441541 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:58.441415    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:23:58.441541 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:58.441438    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r" event={"ID":"95a390b7-6abc-440b-9513-5e06d611abd7","Type":"ContainerStarted","Data":"e171796704ff7a25fa106ce9246c005916d6173e7d7227266102595a4dd8dce5"}
Sep 09 14:23:58.442655 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:58.442631    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf" event={"ID":"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9","Type":"ContainerStarted","Data":"4d5f39d5482fcb86a381a1f6e0725c10e6e8a0420aa578f94f2424edefa3fc1a"}
Sep 09 14:23:58.458876 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:23:58.458828    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r" podStartSLOduration=1.458817293 podStartE2EDuration="1.458817293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:23:58.457718216 +0000 UTC m=+772.220513456" watchObservedRunningTime="2026-09-09 14:23:58.458817293 +0000 UTC m=+772.221612532"
Sep 09 14:24:00.454538 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:00.454502    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf" event={"ID":"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9","Type":"ContainerStarted","Data":"1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946"}
Sep 09 14:24:00.454987 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:00.454605    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:24:00.472971 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:00.472927    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf" podStartSLOduration=2.954049791 podStartE2EDuration="4.4729135s" totalImagesPullingTime="1.518863709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:24:00.471612497 +0000 UTC m=+774.234407738" watchObservedRunningTime="2026-09-09 14:24:00.4729135 +0000 UTC m=+774.235708739"
Sep 09 14:24:04.234280 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.234250    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"]
Sep 09 14:24:04.238168 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.238147    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.240793 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.240771    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ac63e256-serving-cert\""
Sep 09 14:24:04.264389 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.264367    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"]
Sep 09 14:24:04.409805 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.409773    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8l9\" (UniqueName: \"kubernetes.io/projected/7839f660-539f-4ee4-a66d-001f5234eb3e-kube-api-access-2k8l9\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.409921 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.409806    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-metrics-certs\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.409921 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.409836    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7839f660-539f-4ee4-a66d-001f5234eb3e-openshift-service-ca\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.409921 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.409909    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-maas-api-tls\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.511403 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.511332    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8l9\" (UniqueName: \"kubernetes.io/projected/7839f660-539f-4ee4-a66d-001f5234eb3e-kube-api-access-2k8l9\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.511403 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.511364    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-metrics-certs\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.511403 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.511390    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7839f660-539f-4ee4-a66d-001f5234eb3e-openshift-service-ca\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.511620 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.511463    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-maas-api-tls\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.511620 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:24:04.511565    2532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-serving-cert: secret "maas-api-e2e-ait-ac63e256-serving-cert" not found
Sep 09 14:24:04.511708 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:24:04.511638    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-maas-api-tls podName:7839f660-539f-4ee4-a66d-001f5234eb3e nodeName:}" failed. No retries permitted until 2026-09-09 14:24:05.011614905 +0000 UTC m=+778.774410137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-maas-api-tls") pod "maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5" (UID: "7839f660-539f-4ee4-a66d-001f5234eb3e") : secret "maas-api-e2e-ait-ac63e256-serving-cert" not found
Sep 09 14:24:04.512065 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.512049    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7839f660-539f-4ee4-a66d-001f5234eb3e-openshift-service-ca\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.513742 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.513720    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-metrics-certs\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.521891 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.521872    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8l9\" (UniqueName: \"kubernetes.io/projected/7839f660-539f-4ee4-a66d-001f5234eb3e-kube-api-access-2k8l9\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:04.907082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.907012    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"]
Sep 09 14:24:04.910716 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.910694    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:04.913221 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.913201    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ac63e256-dockercfg-zqdp7\""
Sep 09 14:24:04.913343 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.913281    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ac63e256\""
Sep 09 14:24:04.922325 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:04.922303    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"]
Sep 09 14:24:05.016920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.016887    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72657561-6fda-43aa-a3c1-6246e695ee29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf\" (UID: \"72657561-6fda-43aa-a3c1-6246e695ee29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:05.017113 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.016961    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-maas-api-tls\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:05.017113 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.017000    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrm9\" (UniqueName: \"kubernetes.io/projected/72657561-6fda-43aa-a3c1-6246e695ee29-kube-api-access-vmrm9\") pod \"payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf\" (UID: \"72657561-6fda-43aa-a3c1-6246e695ee29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:05.019489 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.019453    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-maas-api-tls\") pod \"maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:05.118351 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.118317    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrm9\" (UniqueName: \"kubernetes.io/projected/72657561-6fda-43aa-a3c1-6246e695ee29-kube-api-access-vmrm9\") pod \"payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf\" (UID: \"72657561-6fda-43aa-a3c1-6246e695ee29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:05.118501 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.118412    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72657561-6fda-43aa-a3c1-6246e695ee29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf\" (UID: \"72657561-6fda-43aa-a3c1-6246e695ee29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:05.119048 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.119030    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72657561-6fda-43aa-a3c1-6246e695ee29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf\" (UID: \"72657561-6fda-43aa-a3c1-6246e695ee29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:05.126472 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.126449    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrm9\" (UniqueName: \"kubernetes.io/projected/72657561-6fda-43aa-a3c1-6246e695ee29-kube-api-access-vmrm9\") pod \"payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf\" (UID: \"72657561-6fda-43aa-a3c1-6246e695ee29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:05.148561 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.148540    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:05.223184 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.223157    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:05.423247 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.423221    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"]
Sep 09 14:24:05.424842 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:24:05.424818    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7839f660_539f_4ee4_a66d_001f5234eb3e.slice/crio-72e6458fa231b40b1f08169992d3a7fab5e63749b240cd9bf06d3c3009b60a3c WatchSource:0}: Error finding container 72e6458fa231b40b1f08169992d3a7fab5e63749b240cd9bf06d3c3009b60a3c: Status 404 returned error can't find the container with id 72e6458fa231b40b1f08169992d3a7fab5e63749b240cd9bf06d3c3009b60a3c
Sep 09 14:24:05.485554 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.485527    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5" event={"ID":"7839f660-539f-4ee4-a66d-001f5234eb3e","Type":"ContainerStarted","Data":"72e6458fa231b40b1f08169992d3a7fab5e63749b240cd9bf06d3c3009b60a3c"}
Sep 09 14:24:05.493807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:05.493782    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"]
Sep 09 14:24:05.495152 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:24:05.495128    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72657561_6fda_43aa_a3c1_6246e695ee29.slice/crio-b145e2051f82c070ced14d0383fcad1622518b55b568558f74ba42cc1da7b30d WatchSource:0}: Error finding container b145e2051f82c070ced14d0383fcad1622518b55b568558f74ba42cc1da7b30d: Status 404 returned error can't find the container with id b145e2051f82c070ced14d0383fcad1622518b55b568558f74ba42cc1da7b30d
Sep 09 14:24:06.465756 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:06.465725    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:24:06.493775 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:06.493743    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5" event={"ID":"7839f660-539f-4ee4-a66d-001f5234eb3e","Type":"ContainerStarted","Data":"9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6"}
Sep 09 14:24:06.493961 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:06.493841    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:06.495856 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:06.495830    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf" event={"ID":"72657561-6fda-43aa-a3c1-6246e695ee29","Type":"ContainerStarted","Data":"f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45"}
Sep 09 14:24:06.496387 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:06.495875    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf" event={"ID":"72657561-6fda-43aa-a3c1-6246e695ee29","Type":"ContainerStarted","Data":"b145e2051f82c070ced14d0383fcad1622518b55b568558f74ba42cc1da7b30d"}
Sep 09 14:24:06.496387 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:06.495909    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:06.511895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:06.511846    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5" podStartSLOduration=2.089056109 podStartE2EDuration="2.511833144s" totalImagesPullingTime="422.777035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:24:06.510782764 +0000 UTC m=+780.273578008" watchObservedRunningTime="2026-09-09 14:24:06.511833144 +0000 UTC m=+780.274628383"
Sep 09 14:24:06.526901 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:06.526862    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf" podStartSLOduration=2.526849166 podStartE2EDuration="2.526849166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:24:06.525829109 +0000 UTC m=+780.288624377" watchObservedRunningTime="2026-09-09 14:24:06.526849166 +0000 UTC m=+780.289644405"
Sep 09 14:24:09.449510 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:09.449478    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:24:11.854418 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.854382    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"]
Sep 09 14:24:11.860006 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.859982    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:24:11.862323 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.862300    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a2ffe2\""
Sep 09 14:24:11.862433 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.862372    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a2ffe2-dockercfg-jckc4\""
Sep 09 14:24:11.867910 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.867885    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"]
Sep 09 14:24:11.884357 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.884326    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-plugins-config-volume\") pod \"payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs\" (UID: \"13b1e077-f1a0-4e1a-a665-c0a0b41c1861\") " pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:24:11.884539 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.884511    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqzg9\" (UniqueName: \"kubernetes.io/projected/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-kube-api-access-jqzg9\") pod \"payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs\" (UID: \"13b1e077-f1a0-4e1a-a665-c0a0b41c1861\") " pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:24:11.985314 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.985287    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqzg9\" (UniqueName: \"kubernetes.io/projected/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-kube-api-access-jqzg9\") pod \"payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs\" (UID: \"13b1e077-f1a0-4e1a-a665-c0a0b41c1861\") " pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:24:11.985454 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.985349    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-plugins-config-volume\") pod \"payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs\" (UID: \"13b1e077-f1a0-4e1a-a665-c0a0b41c1861\") " pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:24:11.986026 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.986006    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-plugins-config-volume\") pod \"payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs\" (UID: \"13b1e077-f1a0-4e1a-a665-c0a0b41c1861\") " pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:24:11.992929 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:11.992907    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqzg9\" (UniqueName: \"kubernetes.io/projected/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-kube-api-access-jqzg9\") pod \"payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs\" (UID: \"13b1e077-f1a0-4e1a-a665-c0a0b41c1861\") " pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:24:12.173751 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:12.173676    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:24:12.445474 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:12.445447    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"]
Sep 09 14:24:12.447092 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:24:12.447055    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b1e077_f1a0_4e1a_a665_c0a0b41c1861.slice/crio-751a06446f1a2f52441e46655a9a49bb799e090b7aa51903d438afd3d8bd91ea WatchSource:0}: Error finding container 751a06446f1a2f52441e46655a9a49bb799e090b7aa51903d438afd3d8bd91ea: Status 404 returned error can't find the container with id 751a06446f1a2f52441e46655a9a49bb799e090b7aa51903d438afd3d8bd91ea
Sep 09 14:24:12.507356 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:12.507336    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:12.527427 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:12.527395    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs" event={"ID":"13b1e077-f1a0-4e1a-a665-c0a0b41c1861","Type":"ContainerStarted","Data":"3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699"}
Sep 09 14:24:12.527545 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:12.527461    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:24:12.527545 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:12.527486    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs" event={"ID":"13b1e077-f1a0-4e1a-a665-c0a0b41c1861","Type":"ContainerStarted","Data":"751a06446f1a2f52441e46655a9a49bb799e090b7aa51903d438afd3d8bd91ea"}
Sep 09 14:24:12.549660 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:12.549594    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs" podStartSLOduration=1.5495758130000001 podStartE2EDuration="1.549575813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:24:12.546067938 +0000 UTC m=+786.308863180" watchObservedRunningTime="2026-09-09 14:24:12.549575813 +0000 UTC m=+786.312371054"
Sep 09 14:24:17.503293 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:17.503260    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:22.592655 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.592616    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"]
Sep 09 14:24:22.596392 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.596369    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.599196 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.599176    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-5cac3324-kserve-self-signed-certs\""
Sep 09 14:24:22.606000 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.605975    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"]
Sep 09 14:24:22.682005 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.681980    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-dshm\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.682123 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.682023    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tmp-dir\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.682123 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.682086    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-home\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.682208 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.682156    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tls-certs\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.682285 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.682205    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2sw4\" (UniqueName: \"kubernetes.io/projected/4414e41e-4360-4b99-81a4-dbe91c8c77b3-kube-api-access-c2sw4\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.682285 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.682229    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-model-cache\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.783508 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.783469    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tls-certs\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.783668 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.783544    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2sw4\" (UniqueName: \"kubernetes.io/projected/4414e41e-4360-4b99-81a4-dbe91c8c77b3-kube-api-access-c2sw4\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.783668 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.783576    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-model-cache\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.783668 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.783633    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-dshm\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.783853 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.783684    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tmp-dir\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.783853 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.783737    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-home\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.784062 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.784039    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-model-cache\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.784208 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.784091    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-home\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.784208 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.784168    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tmp-dir\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.785724 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.785699    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-dshm\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.785811 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.785756    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tls-certs\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.791147 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.791118    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2sw4\" (UniqueName: \"kubernetes.io/projected/4414e41e-4360-4b99-81a4-dbe91c8c77b3-kube-api-access-c2sw4\") pod \"e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") " pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:22.905705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:22.905643    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:23.183978 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:23.183952    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"]
Sep 09 14:24:23.186118 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:24:23.186078    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4414e41e_4360_4b99_81a4_dbe91c8c77b3.slice/crio-0b5eb7362931fb279cf3dcc4739da196d2d22ac689aef84fb576f98b31cda932 WatchSource:0}: Error finding container 0b5eb7362931fb279cf3dcc4739da196d2d22ac689aef84fb576f98b31cda932: Status 404 returned error can't find the container with id 0b5eb7362931fb279cf3dcc4739da196d2d22ac689aef84fb576f98b31cda932
Sep 09 14:24:23.533785 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:23.533750    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:24:23.583010 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:23.582970    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj" event={"ID":"4414e41e-4360-4b99-81a4-dbe91c8c77b3","Type":"ContainerStarted","Data":"293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc"}
Sep 09 14:24:23.583010 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:23.583010    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj" event={"ID":"4414e41e-4360-4b99-81a4-dbe91c8c77b3","Type":"ContainerStarted","Data":"0b5eb7362931fb279cf3dcc4739da196d2d22ac689aef84fb576f98b31cda932"}
Sep 09 14:24:23.602580 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:23.602531    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj" podStartSLOduration=1.6025175379999999 podStartE2EDuration="1.602517538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:24:23.60103421 +0000 UTC m=+797.363829451" watchObservedRunningTime="2026-09-09 14:24:23.602517538 +0000 UTC m=+797.365312796"
Sep 09 14:24:27.277225 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.277187    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"]
Sep 09 14:24:27.281037 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.281014    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:24:27.283437 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.283414    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d85f87\""
Sep 09 14:24:27.283566 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.283545    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d85f87-dockercfg-jxvqz\""
Sep 09 14:24:27.289898 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.289878    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"]
Sep 09 14:24:27.427983 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.427955    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xtbv\" (UniqueName: \"kubernetes.io/projected/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-kube-api-access-4xtbv\") pod \"payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g\" (UID: \"12634c2a-21aa-4f35-84e3-f58ee9bc07f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:24:27.428126 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.428034    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g\" (UID: \"12634c2a-21aa-4f35-84e3-f58ee9bc07f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:24:27.528868 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.528812    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g\" (UID: \"12634c2a-21aa-4f35-84e3-f58ee9bc07f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:24:27.528990 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.528907    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xtbv\" (UniqueName: \"kubernetes.io/projected/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-kube-api-access-4xtbv\") pod \"payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g\" (UID: \"12634c2a-21aa-4f35-84e3-f58ee9bc07f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:24:27.529520 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.529497    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g\" (UID: \"12634c2a-21aa-4f35-84e3-f58ee9bc07f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:24:27.535913 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.535891    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xtbv\" (UniqueName: \"kubernetes.io/projected/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-kube-api-access-4xtbv\") pod \"payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g\" (UID: \"12634c2a-21aa-4f35-84e3-f58ee9bc07f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:24:27.593263 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.593237    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:24:27.866691 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:27.866665    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"]
Sep 09 14:24:27.868891 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:24:27.868864    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12634c2a_21aa_4f35_84e3_f58ee9bc07f8.slice/crio-15476b258aebfad403ec840966c09e1f34cd355be6611f05dfeafe9d6b46f5fa WatchSource:0}: Error finding container 15476b258aebfad403ec840966c09e1f34cd355be6611f05dfeafe9d6b46f5fa: Status 404 returned error can't find the container with id 15476b258aebfad403ec840966c09e1f34cd355be6611f05dfeafe9d6b46f5fa
Sep 09 14:24:28.610179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:28.610146    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g" event={"ID":"12634c2a-21aa-4f35-84e3-f58ee9bc07f8","Type":"ContainerStarted","Data":"14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27"}
Sep 09 14:24:28.610582 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:28.610197    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g" event={"ID":"12634c2a-21aa-4f35-84e3-f58ee9bc07f8","Type":"ContainerStarted","Data":"15476b258aebfad403ec840966c09e1f34cd355be6611f05dfeafe9d6b46f5fa"}
Sep 09 14:24:28.610582 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:28.610275    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:24:28.632939 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:28.632890    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g" podStartSLOduration=1.632874583 podStartE2EDuration="1.632874583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:24:28.631565036 +0000 UTC m=+802.394360276" watchObservedRunningTime="2026-09-09 14:24:28.632874583 +0000 UTC m=+802.395669877"
Sep 09 14:24:32.906247 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:32.906162    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:32.906247 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:32.906218    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:32.918419 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:32.918395    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:33.646092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:33.646068    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:24:39.618190 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:39.618160    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:24:45.156023 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.155991    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"]
Sep 09 14:24:45.156458 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.156215    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5" podUID="7839f660-539f-4ee4-a66d-001f5234eb3e" containerName="maas-api" containerID="cri-o://9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6" gracePeriod=30
Sep 09 14:24:45.220473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.220437    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"]
Sep 09 14:24:45.220670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.220647    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf" podUID="72657561-6fda-43aa-a3c1-6246e695ee29" containerName="payload-pre-processing" containerID="cri-o://f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45" gracePeriod=30
Sep 09 14:24:45.464746 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.464719    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:45.505585 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.505565    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:45.594795 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.594768    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7839f660-539f-4ee4-a66d-001f5234eb3e-openshift-service-ca\") pod \"7839f660-539f-4ee4-a66d-001f5234eb3e\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") "
Sep 09 14:24:45.594938 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.594830    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k8l9\" (UniqueName: \"kubernetes.io/projected/7839f660-539f-4ee4-a66d-001f5234eb3e-kube-api-access-2k8l9\") pod \"7839f660-539f-4ee4-a66d-001f5234eb3e\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") "
Sep 09 14:24:45.594938 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.594882    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-maas-api-tls\") pod \"7839f660-539f-4ee4-a66d-001f5234eb3e\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") "
Sep 09 14:24:45.594938 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.594924    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-metrics-certs\") pod \"7839f660-539f-4ee4-a66d-001f5234eb3e\" (UID: \"7839f660-539f-4ee4-a66d-001f5234eb3e\") "
Sep 09 14:24:45.595144 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.595089    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7839f660-539f-4ee4-a66d-001f5234eb3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7839f660-539f-4ee4-a66d-001f5234eb3e" (UID: "7839f660-539f-4ee4-a66d-001f5234eb3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:24:45.595244 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.595226    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7839f660-539f-4ee4-a66d-001f5234eb3e-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:24:45.596949 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.596920    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7839f660-539f-4ee4-a66d-001f5234eb3e" (UID: "7839f660-539f-4ee4-a66d-001f5234eb3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:24:45.597032 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.596932    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7839f660-539f-4ee4-a66d-001f5234eb3e" (UID: "7839f660-539f-4ee4-a66d-001f5234eb3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:24:45.597032 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.596983    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7839f660-539f-4ee4-a66d-001f5234eb3e-kube-api-access-2k8l9" (OuterVolumeSpecName: "kube-api-access-2k8l9") pod "7839f660-539f-4ee4-a66d-001f5234eb3e" (UID: "7839f660-539f-4ee4-a66d-001f5234eb3e"). InnerVolumeSpecName "kube-api-access-2k8l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:24:45.695660 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.695601    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmrm9\" (UniqueName: \"kubernetes.io/projected/72657561-6fda-43aa-a3c1-6246e695ee29-kube-api-access-vmrm9\") pod \"72657561-6fda-43aa-a3c1-6246e695ee29\" (UID: \"72657561-6fda-43aa-a3c1-6246e695ee29\") "
Sep 09 14:24:45.695660 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.695648    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72657561-6fda-43aa-a3c1-6246e695ee29-plugins-config-volume\") pod \"72657561-6fda-43aa-a3c1-6246e695ee29\" (UID: \"72657561-6fda-43aa-a3c1-6246e695ee29\") "
Sep 09 14:24:45.695903 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.695882    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:24:45.695903 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.695905    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k8l9\" (UniqueName: \"kubernetes.io/projected/7839f660-539f-4ee4-a66d-001f5234eb3e-kube-api-access-2k8l9\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:24:45.696063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.695914    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7839f660-539f-4ee4-a66d-001f5234eb3e-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:24:45.696063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.696000    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72657561-6fda-43aa-a3c1-6246e695ee29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72657561-6fda-43aa-a3c1-6246e695ee29" (UID: "72657561-6fda-43aa-a3c1-6246e695ee29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:24:45.696780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.696755    2532 generic.go:356] "Generic (PLEG): container finished" podID="7839f660-539f-4ee4-a66d-001f5234eb3e" containerID="9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6" exitCode=0
Sep 09 14:24:45.696855 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.696790    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"
Sep 09 14:24:45.696855 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.696834    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5" event={"ID":"7839f660-539f-4ee4-a66d-001f5234eb3e","Type":"ContainerDied","Data":"9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6"}
Sep 09 14:24:45.696969 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.696894    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5" event={"ID":"7839f660-539f-4ee4-a66d-001f5234eb3e","Type":"ContainerDied","Data":"72e6458fa231b40b1f08169992d3a7fab5e63749b240cd9bf06d3c3009b60a3c"}
Sep 09 14:24:45.696969 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.696927    2532 scope.go:117] "RemoveContainer" containerID="9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6"
Sep 09 14:24:45.698173 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.697974    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72657561-6fda-43aa-a3c1-6246e695ee29-kube-api-access-vmrm9" (OuterVolumeSpecName: "kube-api-access-vmrm9") pod "72657561-6fda-43aa-a3c1-6246e695ee29" (UID: "72657561-6fda-43aa-a3c1-6246e695ee29"). InnerVolumeSpecName "kube-api-access-vmrm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:24:45.698414 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.698389    2532 generic.go:356] "Generic (PLEG): container finished" podID="72657561-6fda-43aa-a3c1-6246e695ee29" containerID="f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45" exitCode=0
Sep 09 14:24:45.698516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.698419    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf" event={"ID":"72657561-6fda-43aa-a3c1-6246e695ee29","Type":"ContainerDied","Data":"f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45"}
Sep 09 14:24:45.698516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.698434    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"
Sep 09 14:24:45.698516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.698469    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf" event={"ID":"72657561-6fda-43aa-a3c1-6246e695ee29","Type":"ContainerDied","Data":"b145e2051f82c070ced14d0383fcad1622518b55b568558f74ba42cc1da7b30d"}
Sep 09 14:24:45.707174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.707153    2532 scope.go:117] "RemoveContainer" containerID="9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6"
Sep 09 14:24:45.707438 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:24:45.707418    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6\": container with ID starting with 9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6 not found: ID does not exist" containerID="9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6"
Sep 09 14:24:45.707518 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.707448    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6"} err="failed to get container status \"9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6\": rpc error: code = NotFound desc = could not find container \"9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6\": container with ID starting with 9570b15141a5f78048d069a0fae28c3ca8fa7e956d4730f244ec989118402bc6 not found: ID does not exist"
Sep 09 14:24:45.707518 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.707473    2532 scope.go:117] "RemoveContainer" containerID="f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45"
Sep 09 14:24:45.715024 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.715008    2532 scope.go:117] "RemoveContainer" containerID="f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45"
Sep 09 14:24:45.715310 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:24:45.715292    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45\": container with ID starting with f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45 not found: ID does not exist" containerID="f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45"
Sep 09 14:24:45.715364 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.715317    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45"} err="failed to get container status \"f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45\": rpc error: code = NotFound desc = could not find container \"f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45\": container with ID starting with f08e9f0d1a02fecc7f2d5034f09eb225f97536a1d52b7c630900667d272bcd45 not found: ID does not exist"
Sep 09 14:24:45.719865 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.719844    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"]
Sep 09 14:24:45.724228 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.724209    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac63e256-7f7575d4d9-9slv5"]
Sep 09 14:24:45.734481 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.734461    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"]
Sep 09 14:24:45.738426 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.738407    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac63e256-76db65cf66-smgkf"]
Sep 09 14:24:45.797114 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.797072    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmrm9\" (UniqueName: \"kubernetes.io/projected/72657561-6fda-43aa-a3c1-6246e695ee29-kube-api-access-vmrm9\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:24:45.797114 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:45.797113    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72657561-6fda-43aa-a3c1-6246e695ee29-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:24:46.736343 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:46.736313    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72657561-6fda-43aa-a3c1-6246e695ee29" path="/var/lib/kubelet/pods/72657561-6fda-43aa-a3c1-6246e695ee29/volumes"
Sep 09 14:24:46.736748 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:46.736733    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7839f660-539f-4ee4-a66d-001f5234eb3e" path="/var/lib/kubelet/pods/7839f660-539f-4ee4-a66d-001f5234eb3e/volumes"
Sep 09 14:24:49.538896 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.538864    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"]
Sep 09 14:24:49.539390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.539367    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72657561-6fda-43aa-a3c1-6246e695ee29" containerName="payload-pre-processing"
Sep 09 14:24:49.539390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.539386    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="72657561-6fda-43aa-a3c1-6246e695ee29" containerName="payload-pre-processing"
Sep 09 14:24:49.539540 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.539408    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7839f660-539f-4ee4-a66d-001f5234eb3e" containerName="maas-api"
Sep 09 14:24:49.539540 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.539417    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="7839f660-539f-4ee4-a66d-001f5234eb3e" containerName="maas-api"
Sep 09 14:24:49.539658 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.539555    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="72657561-6fda-43aa-a3c1-6246e695ee29" containerName="payload-pre-processing"
Sep 09 14:24:49.539658 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.539569    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="7839f660-539f-4ee4-a66d-001f5234eb3e" containerName="maas-api"
Sep 09 14:24:49.544623 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.544605    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.547034 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.547008    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3e00fb33-serving-cert\""
Sep 09 14:24:49.553781 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.553761    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"]
Sep 09 14:24:49.626357 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.626331    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-maas-api-tls\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.626453 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.626363    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60347bea-9416-4ccd-9af9-5c06f2ddb609-openshift-service-ca\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.626453 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.626388    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-metrics-certs\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.626544 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.626490    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w828m\" (UniqueName: \"kubernetes.io/projected/60347bea-9416-4ccd-9af9-5c06f2ddb609-kube-api-access-w828m\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.727004 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.726969    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-maas-api-tls\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.727179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.727016    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60347bea-9416-4ccd-9af9-5c06f2ddb609-openshift-service-ca\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.727179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.727059    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-metrics-certs\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.727302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.727175    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w828m\" (UniqueName: \"kubernetes.io/projected/60347bea-9416-4ccd-9af9-5c06f2ddb609-kube-api-access-w828m\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.727923 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.727892    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60347bea-9416-4ccd-9af9-5c06f2ddb609-openshift-service-ca\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.729470 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.729443    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-metrics-certs\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.729767 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.729740    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-maas-api-tls\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.737663 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.737637    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w828m\" (UniqueName: \"kubernetes.io/projected/60347bea-9416-4ccd-9af9-5c06f2ddb609-kube-api-access-w828m\") pod \"maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:49.856996 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:49.856906    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:50.135393 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:50.135367    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"]
Sep 09 14:24:50.137297 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:24:50.137272    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60347bea_9416_4ccd_9af9_5c06f2ddb609.slice/crio-d6d6994d711b6f5c253e073a1688f87aaab7f800a8bf4b1aea082eacd9060b7e WatchSource:0}: Error finding container d6d6994d711b6f5c253e073a1688f87aaab7f800a8bf4b1aea082eacd9060b7e: Status 404 returned error can't find the container with id d6d6994d711b6f5c253e073a1688f87aaab7f800a8bf4b1aea082eacd9060b7e
Sep 09 14:24:50.736531 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:50.736498    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj" event={"ID":"60347bea-9416-4ccd-9af9-5c06f2ddb609","Type":"ContainerStarted","Data":"d6d6994d711b6f5c253e073a1688f87aaab7f800a8bf4b1aea082eacd9060b7e"}
Sep 09 14:24:51.743597 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:51.743559    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj" event={"ID":"60347bea-9416-4ccd-9af9-5c06f2ddb609","Type":"ContainerStarted","Data":"aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc"}
Sep 09 14:24:51.743962 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:51.743666    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:24:51.762069 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:51.762013    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj" podStartSLOduration=2.104925259 podStartE2EDuration="2.761993048s" totalImagesPullingTime="657.067789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:24:51.761489217 +0000 UTC m=+825.524284459" watchObservedRunningTime="2026-09-09 14:24:51.761993048 +0000 UTC m=+825.524788289"
Sep 09 14:24:57.753638 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:24:57.753609    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:25:20.396639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.396606    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"]
Sep 09 14:25:20.397157 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.396842    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj" podUID="60347bea-9416-4ccd-9af9-5c06f2ddb609" containerName="maas-api" containerID="cri-o://aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc" gracePeriod=30
Sep 09 14:25:20.647233 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.647182    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:25:20.806685 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.806654    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-metrics-certs\") pod \"60347bea-9416-4ccd-9af9-5c06f2ddb609\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") "
Sep 09 14:25:20.806844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.806777    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w828m\" (UniqueName: \"kubernetes.io/projected/60347bea-9416-4ccd-9af9-5c06f2ddb609-kube-api-access-w828m\") pod \"60347bea-9416-4ccd-9af9-5c06f2ddb609\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") "
Sep 09 14:25:20.806844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.806809    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60347bea-9416-4ccd-9af9-5c06f2ddb609-openshift-service-ca\") pod \"60347bea-9416-4ccd-9af9-5c06f2ddb609\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") "
Sep 09 14:25:20.806844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.806835    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-maas-api-tls\") pod \"60347bea-9416-4ccd-9af9-5c06f2ddb609\" (UID: \"60347bea-9416-4ccd-9af9-5c06f2ddb609\") "
Sep 09 14:25:20.807181 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.807145    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60347bea-9416-4ccd-9af9-5c06f2ddb609-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60347bea-9416-4ccd-9af9-5c06f2ddb609" (UID: "60347bea-9416-4ccd-9af9-5c06f2ddb609"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:25:20.809045 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.809000    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60347bea-9416-4ccd-9af9-5c06f2ddb609-kube-api-access-w828m" (OuterVolumeSpecName: "kube-api-access-w828m") pod "60347bea-9416-4ccd-9af9-5c06f2ddb609" (UID: "60347bea-9416-4ccd-9af9-5c06f2ddb609"). InnerVolumeSpecName "kube-api-access-w828m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:25:20.809254 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.809222    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "60347bea-9416-4ccd-9af9-5c06f2ddb609" (UID: "60347bea-9416-4ccd-9af9-5c06f2ddb609"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:25:20.809254 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.809243    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60347bea-9416-4ccd-9af9-5c06f2ddb609" (UID: "60347bea-9416-4ccd-9af9-5c06f2ddb609"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:25:20.893963 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.893934    2532 generic.go:356] "Generic (PLEG): container finished" podID="60347bea-9416-4ccd-9af9-5c06f2ddb609" containerID="aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc" exitCode=0
Sep 09 14:25:20.894080 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.893970    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj" event={"ID":"60347bea-9416-4ccd-9af9-5c06f2ddb609","Type":"ContainerDied","Data":"aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc"}
Sep 09 14:25:20.894080 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.893984    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"
Sep 09 14:25:20.894080 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.894023    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj" event={"ID":"60347bea-9416-4ccd-9af9-5c06f2ddb609","Type":"ContainerDied","Data":"d6d6994d711b6f5c253e073a1688f87aaab7f800a8bf4b1aea082eacd9060b7e"}
Sep 09 14:25:20.894080 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.894046    2532 scope.go:117] "RemoveContainer" containerID="aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc"
Sep 09 14:25:20.905427 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.905408    2532 scope.go:117] "RemoveContainer" containerID="aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc"
Sep 09 14:25:20.905702 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:25:20.905686    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc\": container with ID starting with aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc not found: ID does not exist" containerID="aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc"
Sep 09 14:25:20.905759 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.905709    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc"} err="failed to get container status \"aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc\": rpc error: code = NotFound desc = could not find container \"aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc\": container with ID starting with aa1d1ccf0bf0fcd8259be2be7b7760b9ab3f6d8f2e741334b44cc7f0e05d30cc not found: ID does not exist"
Sep 09 14:25:20.907910 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.907895    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w828m\" (UniqueName: \"kubernetes.io/projected/60347bea-9416-4ccd-9af9-5c06f2ddb609-kube-api-access-w828m\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:25:20.907968 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.907912    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60347bea-9416-4ccd-9af9-5c06f2ddb609-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:25:20.907968 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.907922    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:25:20.907968 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.907930    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60347bea-9416-4ccd-9af9-5c06f2ddb609-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:25:20.918075 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.918054    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"]
Sep 09 14:25:20.922770 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:20.922745    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3e00fb33-5bb5cd9859-qxftj"]
Sep 09 14:25:22.734938 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:22.734896    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60347bea-9416-4ccd-9af9-5c06f2ddb609" path="/var/lib/kubelet/pods/60347bea-9416-4ccd-9af9-5c06f2ddb609/volumes"
Sep 09 14:25:23.798867 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.798829    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"]
Sep 09 14:25:23.799854 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.799830    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60347bea-9416-4ccd-9af9-5c06f2ddb609" containerName="maas-api"
Sep 09 14:25:23.799854 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.799857    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="60347bea-9416-4ccd-9af9-5c06f2ddb609" containerName="maas-api"
Sep 09 14:25:23.800110 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.800079    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="60347bea-9416-4ccd-9af9-5c06f2ddb609" containerName="maas-api"
Sep 09 14:25:23.805487 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.805467    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:23.808205 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.808184    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-5cac3324-kserve-self-signed-certs\""
Sep 09 14:25:23.809932 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.809910    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"]
Sep 09 14:25:23.935146 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.935088    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-dshm\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:23.935302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.935156    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-home\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:23.935302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.935212    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9tm\" (UniqueName: \"kubernetes.io/projected/530702a5-679b-4479-9535-d67731cb227a-kube-api-access-ct9tm\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:23.935302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.935252    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/530702a5-679b-4479-9535-d67731cb227a-tls-certs\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:23.935460 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.935338    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-model-cache\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:23.935460 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:23.935370    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-tmp-dir\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.036446 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.036416    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-tmp-dir\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.036628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.036467    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-dshm\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.036628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.036503    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-home\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.036628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.036526    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9tm\" (UniqueName: \"kubernetes.io/projected/530702a5-679b-4479-9535-d67731cb227a-kube-api-access-ct9tm\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.036628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.036553    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/530702a5-679b-4479-9535-d67731cb227a-tls-certs\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.036628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.036616    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-model-cache\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.036912 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.036860    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-tmp-dir\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.036912 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.036895    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-home\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.036985 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.036923    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-model-cache\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.038733 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.038715    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-dshm\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.039001 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.038980    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/530702a5-679b-4479-9535-d67731cb227a-tls-certs\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.050047 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.049992    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9tm\" (UniqueName: \"kubernetes.io/projected/530702a5-679b-4479-9535-d67731cb227a-kube-api-access-ct9tm\") pod \"e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") " pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.115899 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.115879    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:24.391321 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.391298    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"]
Sep 09 14:25:24.393215 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:25:24.393190    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod530702a5_679b_4479_9535_d67731cb227a.slice/crio-4c6b402c08074a91e1585a41b89dcc32fc07338cd776e1a37868feb777b4a310 WatchSource:0}: Error finding container 4c6b402c08074a91e1585a41b89dcc32fc07338cd776e1a37868feb777b4a310: Status 404 returned error can't find the container with id 4c6b402c08074a91e1585a41b89dcc32fc07338cd776e1a37868feb777b4a310
Sep 09 14:25:24.892819 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.892789    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"]
Sep 09 14:25:24.897204 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.897176    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:24.899649 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.899630    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-701ccc59-serving-cert\""
Sep 09 14:25:24.907239 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.907167    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"]
Sep 09 14:25:24.918713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.918685    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c" event={"ID":"530702a5-679b-4479-9535-d67731cb227a","Type":"ContainerStarted","Data":"d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536"}
Sep 09 14:25:24.918839 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.918728    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c" event={"ID":"530702a5-679b-4479-9535-d67731cb227a","Type":"ContainerStarted","Data":"4c6b402c08074a91e1585a41b89dcc32fc07338cd776e1a37868feb777b4a310"}
Sep 09 14:25:24.936634 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:24.936588    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c" podStartSLOduration=1.936575186 podStartE2EDuration="1.936575186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:25:24.935022104 +0000 UTC m=+858.697817346" watchObservedRunningTime="2026-09-09 14:25:24.936575186 +0000 UTC m=+858.699370425"
Sep 09 14:25:25.047723 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.047693    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-metrics-certs\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.047872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.047848    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84696909-0d85-4ac5-b680-e499bae27380-openshift-service-ca\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.047997 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.047970    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szk5x\" (UniqueName: \"kubernetes.io/projected/84696909-0d85-4ac5-b680-e499bae27380-kube-api-access-szk5x\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.048043 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.048029    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-maas-api-tls\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.148902 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.148845    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-metrics-certs\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.149020 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.148907    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84696909-0d85-4ac5-b680-e499bae27380-openshift-service-ca\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.149020 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.148952    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szk5x\" (UniqueName: \"kubernetes.io/projected/84696909-0d85-4ac5-b680-e499bae27380-kube-api-access-szk5x\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.149020 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.148973    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-maas-api-tls\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.149618 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.149595    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84696909-0d85-4ac5-b680-e499bae27380-openshift-service-ca\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.151118 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.151075    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-maas-api-tls\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.151316 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.151298    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-metrics-certs\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.156644 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.156624    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szk5x\" (UniqueName: \"kubernetes.io/projected/84696909-0d85-4ac5-b680-e499bae27380-kube-api-access-szk5x\") pod \"maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.211799 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.211776    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:25.486443 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.486415    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"]
Sep 09 14:25:25.488151 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:25:25.488126    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84696909_0d85_4ac5_b680_e499bae27380.slice/crio-171fa8a97eb075a37e0700df87a740a4e6d0630f6214cae46c8517be2f5435ea WatchSource:0}: Error finding container 171fa8a97eb075a37e0700df87a740a4e6d0630f6214cae46c8517be2f5435ea: Status 404 returned error can't find the container with id 171fa8a97eb075a37e0700df87a740a4e6d0630f6214cae46c8517be2f5435ea
Sep 09 14:25:25.926035 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:25.925942    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck" event={"ID":"84696909-0d85-4ac5-b680-e499bae27380","Type":"ContainerStarted","Data":"171fa8a97eb075a37e0700df87a740a4e6d0630f6214cae46c8517be2f5435ea"}
Sep 09 14:25:26.931845 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:26.931765    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck" event={"ID":"84696909-0d85-4ac5-b680-e499bae27380","Type":"ContainerStarted","Data":"4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd"}
Sep 09 14:25:26.932187 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:26.931928    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:26.950497 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:26.950448    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck" podStartSLOduration=1.820338344 podStartE2EDuration="2.950429379s" totalImagesPullingTime="1.130091035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:25:26.948382759 +0000 UTC m=+860.711177998" watchObservedRunningTime="2026-09-09 14:25:26.950429379 +0000 UTC m=+860.713224620"
Sep 09 14:25:32.941439 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:32.941413    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:34.116665 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:34.116613    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:34.117031 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:34.116687    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:34.129166 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:34.129139    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:34.985240 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:34.985211    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:25:55.600288 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.592574    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"]
Sep 09 14:25:55.600288 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.592903    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck" podUID="84696909-0d85-4ac5-b680-e499bae27380" containerName="maas-api" containerID="cri-o://4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd" gracePeriod=30
Sep 09 14:25:55.852685 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.852630    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:55.917754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.917725    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-maas-api-tls\") pod \"84696909-0d85-4ac5-b680-e499bae27380\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") "
Sep 09 14:25:55.917871 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.917789    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84696909-0d85-4ac5-b680-e499bae27380-openshift-service-ca\") pod \"84696909-0d85-4ac5-b680-e499bae27380\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") "
Sep 09 14:25:55.917946 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.917931    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-metrics-certs\") pod \"84696909-0d85-4ac5-b680-e499bae27380\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") "
Sep 09 14:25:55.918014 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.918001    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szk5x\" (UniqueName: \"kubernetes.io/projected/84696909-0d85-4ac5-b680-e499bae27380-kube-api-access-szk5x\") pod \"84696909-0d85-4ac5-b680-e499bae27380\" (UID: \"84696909-0d85-4ac5-b680-e499bae27380\") "
Sep 09 14:25:55.918429 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.918183    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84696909-0d85-4ac5-b680-e499bae27380-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84696909-0d85-4ac5-b680-e499bae27380" (UID: "84696909-0d85-4ac5-b680-e499bae27380"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:25:55.918429 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.918400    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84696909-0d85-4ac5-b680-e499bae27380-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:25:55.920008 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.919979    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84696909-0d85-4ac5-b680-e499bae27380" (UID: "84696909-0d85-4ac5-b680-e499bae27380"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:25:55.920149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.920031    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "84696909-0d85-4ac5-b680-e499bae27380" (UID: "84696909-0d85-4ac5-b680-e499bae27380"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:25:55.920149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:55.920057    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84696909-0d85-4ac5-b680-e499bae27380-kube-api-access-szk5x" (OuterVolumeSpecName: "kube-api-access-szk5x") pod "84696909-0d85-4ac5-b680-e499bae27380" (UID: "84696909-0d85-4ac5-b680-e499bae27380"). InnerVolumeSpecName "kube-api-access-szk5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:25:56.019032 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.018991    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:25:56.019032 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.019026    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szk5x\" (UniqueName: \"kubernetes.io/projected/84696909-0d85-4ac5-b680-e499bae27380-kube-api-access-szk5x\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:25:56.019032 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.019040    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84696909-0d85-4ac5-b680-e499bae27380-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:25:56.079971 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.079940    2532 generic.go:356] "Generic (PLEG): container finished" podID="84696909-0d85-4ac5-b680-e499bae27380" containerID="4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd" exitCode=0
Sep 09 14:25:56.080181 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.079999    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"
Sep 09 14:25:56.080181 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.080025    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck" event={"ID":"84696909-0d85-4ac5-b680-e499bae27380","Type":"ContainerDied","Data":"4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd"}
Sep 09 14:25:56.080181 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.080075    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck" event={"ID":"84696909-0d85-4ac5-b680-e499bae27380","Type":"ContainerDied","Data":"171fa8a97eb075a37e0700df87a740a4e6d0630f6214cae46c8517be2f5435ea"}
Sep 09 14:25:56.080181 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.080108    2532 scope.go:117] "RemoveContainer" containerID="4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd"
Sep 09 14:25:56.089326 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.089308    2532 scope.go:117] "RemoveContainer" containerID="4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd"
Sep 09 14:25:56.089587 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:25:56.089567    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd\": container with ID starting with 4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd not found: ID does not exist" containerID="4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd"
Sep 09 14:25:56.089658 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.089595    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd"} err="failed to get container status \"4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd\": rpc error: code = NotFound desc = could not find container \"4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd\": container with ID starting with 4c6898032002ab55f15a5be886d96291631c15a81a34f08aefcc92ccf982eacd not found: ID does not exist"
Sep 09 14:25:56.102435 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.102405    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"]
Sep 09 14:25:56.106009 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.105959    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-701ccc59-64fdff76bf-gwhck"]
Sep 09 14:25:56.736560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:25:56.736526    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84696909-0d85-4ac5-b680-e499bae27380" path="/var/lib/kubelet/pods/84696909-0d85-4ac5-b680-e499bae27380/volumes"
Sep 09 14:26:01.397301 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.397263    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"]
Sep 09 14:26:01.397955 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.397934    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84696909-0d85-4ac5-b680-e499bae27380" containerName="maas-api"
Sep 09 14:26:01.398004 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.397959    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="84696909-0d85-4ac5-b680-e499bae27380" containerName="maas-api"
Sep 09 14:26:01.398149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.398135    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="84696909-0d85-4ac5-b680-e499bae27380" containerName="maas-api"
Sep 09 14:26:01.403244 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.403223    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.405495 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.405474    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a80b9686-serving-cert\""
Sep 09 14:26:01.414236 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.414215    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"]
Sep 09 14:26:01.467574 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.467551    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-metrics-certs\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.467705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.467624    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cs65\" (UniqueName: \"kubernetes.io/projected/fb0ab440-e463-4359-8e8a-2264ecfccc3e-kube-api-access-9cs65\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.467773 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.467701    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-maas-api-tls\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.467773 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.467760    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0ab440-e463-4359-8e8a-2264ecfccc3e-openshift-service-ca\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.569014 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.568988    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cs65\" (UniqueName: \"kubernetes.io/projected/fb0ab440-e463-4359-8e8a-2264ecfccc3e-kube-api-access-9cs65\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.569153 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.569030    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-maas-api-tls\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.569153 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.569070    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0ab440-e463-4359-8e8a-2264ecfccc3e-openshift-service-ca\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.569284 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.569157    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-metrics-certs\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.569284 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:26:01.569182    2532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-serving-cert: secret "maas-api-e2e-derive-a80b9686-serving-cert" not found
Sep 09 14:26:01.569284 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:26:01.569249    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-maas-api-tls podName:fb0ab440-e463-4359-8e8a-2264ecfccc3e nodeName:}" failed. No retries permitted until 2026-09-09 14:26:02.06922723 +0000 UTC m=+895.832022454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-maas-api-tls") pod "maas-api-e2e-derive-a80b9686-76947f6649-r4l9g" (UID: "fb0ab440-e463-4359-8e8a-2264ecfccc3e") : secret "maas-api-e2e-derive-a80b9686-serving-cert" not found
Sep 09 14:26:01.569703 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.569678    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0ab440-e463-4359-8e8a-2264ecfccc3e-openshift-service-ca\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.571426 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.571406    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-metrics-certs\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:01.586432 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:01.586404    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cs65\" (UniqueName: \"kubernetes.io/projected/fb0ab440-e463-4359-8e8a-2264ecfccc3e-kube-api-access-9cs65\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:02.074428 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:02.074391    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-maas-api-tls\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:02.077176 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:02.077152    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-maas-api-tls\") pod \"maas-api-e2e-derive-a80b9686-76947f6649-r4l9g\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:02.315187 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:02.315146    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:02.588555 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:02.588526    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"]
Sep 09 14:26:02.590402 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:26:02.590375    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0ab440_e463_4359_8e8a_2264ecfccc3e.slice/crio-1f07b0f2ff130f51b34ece5123bcf58bb89fe381d957b4851614a2353d521d82 WatchSource:0}: Error finding container 1f07b0f2ff130f51b34ece5123bcf58bb89fe381d957b4851614a2353d521d82: Status 404 returned error can't find the container with id 1f07b0f2ff130f51b34ece5123bcf58bb89fe381d957b4851614a2353d521d82
Sep 09 14:26:03.119685 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:03.119649    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g" event={"ID":"fb0ab440-e463-4359-8e8a-2264ecfccc3e","Type":"ContainerStarted","Data":"1f07b0f2ff130f51b34ece5123bcf58bb89fe381d957b4851614a2353d521d82"}
Sep 09 14:26:04.127302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:04.127214    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g" event={"ID":"fb0ab440-e463-4359-8e8a-2264ecfccc3e","Type":"ContainerStarted","Data":"20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316"}
Sep 09 14:26:04.127302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:04.127300    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:04.146002 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:04.145953    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g" podStartSLOduration=1.9270447050000001 podStartE2EDuration="3.14594147s" totalImagesPullingTime="1.218896765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:26:04.145155601 +0000 UTC m=+897.907950841" watchObservedRunningTime="2026-09-09 14:26:04.14594147 +0000 UTC m=+897.908736709"
Sep 09 14:26:06.700796 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:06.700768    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:26:06.701233 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:06.700775    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:26:10.138182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:10.138157    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:21.905632 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:21.905594    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"]
Sep 09 14:26:22.492612 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:22.492576    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"]
Sep 09 14:26:31.835350 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:31.835300    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"]
Sep 09 14:26:31.835943 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:31.835511    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g" podUID="fb0ab440-e463-4359-8e8a-2264ecfccc3e" containerName="maas-api" containerID="cri-o://20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316" gracePeriod=30
Sep 09 14:26:32.089390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.089338    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:32.161467 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.161441    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-maas-api-tls\") pod \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") "
Sep 09 14:26:32.161605 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.161489    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0ab440-e463-4359-8e8a-2264ecfccc3e-openshift-service-ca\") pod \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") "
Sep 09 14:26:32.161605 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.161530    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cs65\" (UniqueName: \"kubernetes.io/projected/fb0ab440-e463-4359-8e8a-2264ecfccc3e-kube-api-access-9cs65\") pod \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") "
Sep 09 14:26:32.161738 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.161625    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-metrics-certs\") pod \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\" (UID: \"fb0ab440-e463-4359-8e8a-2264ecfccc3e\") "
Sep 09 14:26:32.161880 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.161852    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0ab440-e463-4359-8e8a-2264ecfccc3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb0ab440-e463-4359-8e8a-2264ecfccc3e" (UID: "fb0ab440-e463-4359-8e8a-2264ecfccc3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:26:32.162010 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.161995    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0ab440-e463-4359-8e8a-2264ecfccc3e-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:32.163594 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.163563    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb0ab440-e463-4359-8e8a-2264ecfccc3e" (UID: "fb0ab440-e463-4359-8e8a-2264ecfccc3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:26:32.163680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.163627    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fb0ab440-e463-4359-8e8a-2264ecfccc3e" (UID: "fb0ab440-e463-4359-8e8a-2264ecfccc3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:26:32.163718 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.163704    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0ab440-e463-4359-8e8a-2264ecfccc3e-kube-api-access-9cs65" (OuterVolumeSpecName: "kube-api-access-9cs65") pod "fb0ab440-e463-4359-8e8a-2264ecfccc3e" (UID: "fb0ab440-e463-4359-8e8a-2264ecfccc3e"). InnerVolumeSpecName "kube-api-access-9cs65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:26:32.263159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.263094    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:32.263159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.263149    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cs65\" (UniqueName: \"kubernetes.io/projected/fb0ab440-e463-4359-8e8a-2264ecfccc3e-kube-api-access-9cs65\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:32.263363 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.263164    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0ab440-e463-4359-8e8a-2264ecfccc3e-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:32.272465 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.272439    2532 generic.go:356] "Generic (PLEG): container finished" podID="fb0ab440-e463-4359-8e8a-2264ecfccc3e" containerID="20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316" exitCode=0
Sep 09 14:26:32.272570 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.272477    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"
Sep 09 14:26:32.272570 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.272514    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g" event={"ID":"fb0ab440-e463-4359-8e8a-2264ecfccc3e","Type":"ContainerDied","Data":"20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316"}
Sep 09 14:26:32.272570 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.272563    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g" event={"ID":"fb0ab440-e463-4359-8e8a-2264ecfccc3e","Type":"ContainerDied","Data":"1f07b0f2ff130f51b34ece5123bcf58bb89fe381d957b4851614a2353d521d82"}
Sep 09 14:26:32.272703 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.272584    2532 scope.go:117] "RemoveContainer" containerID="20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316"
Sep 09 14:26:32.286009 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.285979    2532 scope.go:117] "RemoveContainer" containerID="20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316"
Sep 09 14:26:32.286370 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:26:32.286343    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316\": container with ID starting with 20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316 not found: ID does not exist" containerID="20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316"
Sep 09 14:26:32.286460 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.286381    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316"} err="failed to get container status \"20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316\": rpc error: code = NotFound desc = could not find container \"20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316\": container with ID starting with 20c594add111dbece5882e8ee4fe2856000cd89d3ebbe4b99d8995e80c119316 not found: ID does not exist"
Sep 09 14:26:32.297651 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.297623    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"]
Sep 09 14:26:32.300836 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.300809    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a80b9686-76947f6649-r4l9g"]
Sep 09 14:26:32.735063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:32.735036    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb0ab440-e463-4359-8e8a-2264ecfccc3e" path="/var/lib/kubelet/pods/fb0ab440-e463-4359-8e8a-2264ecfccc3e/volumes"
Sep 09 14:26:34.648466 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:34.648434    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"]
Sep 09 14:26:34.648861 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:34.648664    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs" podUID="13b1e077-f1a0-4e1a-a665-c0a0b41c1861" containerName="payload-processing" containerID="cri-o://3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699" gracePeriod=30
Sep 09 14:26:34.900481 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:34.900422    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:26:34.988487 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:34.988461    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-plugins-config-volume\") pod \"13b1e077-f1a0-4e1a-a665-c0a0b41c1861\" (UID: \"13b1e077-f1a0-4e1a-a665-c0a0b41c1861\") "
Sep 09 14:26:34.988619 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:34.988511    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqzg9\" (UniqueName: \"kubernetes.io/projected/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-kube-api-access-jqzg9\") pod \"13b1e077-f1a0-4e1a-a665-c0a0b41c1861\" (UID: \"13b1e077-f1a0-4e1a-a665-c0a0b41c1861\") "
Sep 09 14:26:34.988829 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:34.988805    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13b1e077-f1a0-4e1a-a665-c0a0b41c1861" (UID: "13b1e077-f1a0-4e1a-a665-c0a0b41c1861"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:26:34.990513 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:34.990493    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-kube-api-access-jqzg9" (OuterVolumeSpecName: "kube-api-access-jqzg9") pod "13b1e077-f1a0-4e1a-a665-c0a0b41c1861" (UID: "13b1e077-f1a0-4e1a-a665-c0a0b41c1861"). InnerVolumeSpecName "kube-api-access-jqzg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:26:35.089974 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.089952    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:35.089974 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.089974    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqzg9\" (UniqueName: \"kubernetes.io/projected/13b1e077-f1a0-4e1a-a665-c0a0b41c1861-kube-api-access-jqzg9\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:35.290363 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.290329    2532 generic.go:356] "Generic (PLEG): container finished" podID="13b1e077-f1a0-4e1a-a665-c0a0b41c1861" containerID="3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699" exitCode=0
Sep 09 14:26:35.290513 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.290369    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"
Sep 09 14:26:35.290513 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.290376    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs" event={"ID":"13b1e077-f1a0-4e1a-a665-c0a0b41c1861","Type":"ContainerDied","Data":"3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699"}
Sep 09 14:26:35.290513 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.290425    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs" event={"ID":"13b1e077-f1a0-4e1a-a665-c0a0b41c1861","Type":"ContainerDied","Data":"751a06446f1a2f52441e46655a9a49bb799e090b7aa51903d438afd3d8bd91ea"}
Sep 09 14:26:35.290513 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.290449    2532 scope.go:117] "RemoveContainer" containerID="3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699"
Sep 09 14:26:35.299182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.299165    2532 scope.go:117] "RemoveContainer" containerID="3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699"
Sep 09 14:26:35.299451 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:26:35.299431    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699\": container with ID starting with 3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699 not found: ID does not exist" containerID="3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699"
Sep 09 14:26:35.299500 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.299461    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699"} err="failed to get container status \"3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699\": rpc error: code = NotFound desc = could not find container \"3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699\": container with ID starting with 3e3299793b490f9fafcec2bd1c9e8c11dfef7fdadc0de588193ddf37dd7e9699 not found: ID does not exist"
Sep 09 14:26:35.311608 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.311587    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"]
Sep 09 14:26:35.315970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:35.315948    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a2ffe2-98cf9d6d5-czlgs"]
Sep 09 14:26:36.734540 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:36.734506    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13b1e077-f1a0-4e1a-a665-c0a0b41c1861" path="/var/lib/kubelet/pods/13b1e077-f1a0-4e1a-a665-c0a0b41c1861/volumes"
Sep 09 14:26:36.924463 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:36.924421    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj" podUID="4414e41e-4360-4b99-81a4-dbe91c8c77b3" containerName="main" containerID="cri-o://293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc" gracePeriod=45
Sep 09 14:26:37.179561 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.179536    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:26:37.303397 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.303324    2532 generic.go:356] "Generic (PLEG): container finished" podID="4414e41e-4360-4b99-81a4-dbe91c8c77b3" containerID="293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc" exitCode=0
Sep 09 14:26:37.303397 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.303364    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"
Sep 09 14:26:37.303397 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.303382    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj" event={"ID":"4414e41e-4360-4b99-81a4-dbe91c8c77b3","Type":"ContainerDied","Data":"293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc"}
Sep 09 14:26:37.303619 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.303411    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj" event={"ID":"4414e41e-4360-4b99-81a4-dbe91c8c77b3","Type":"ContainerDied","Data":"0b5eb7362931fb279cf3dcc4739da196d2d22ac689aef84fb576f98b31cda932"}
Sep 09 14:26:37.303619 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.303428    2532 scope.go:117] "RemoveContainer" containerID="293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc"
Sep 09 14:26:37.308637 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.308610    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-dshm\") pod \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") "
Sep 09 14:26:37.308763 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.308678    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tmp-dir\") pod \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") "
Sep 09 14:26:37.308763 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.308721    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2sw4\" (UniqueName: \"kubernetes.io/projected/4414e41e-4360-4b99-81a4-dbe91c8c77b3-kube-api-access-c2sw4\") pod \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") "
Sep 09 14:26:37.308763 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.308750    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-home\") pod \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") "
Sep 09 14:26:37.309089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.308943    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-model-cache\") pod \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") "
Sep 09 14:26:37.309089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.308963    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4414e41e-4360-4b99-81a4-dbe91c8c77b3" (UID: "4414e41e-4360-4b99-81a4-dbe91c8c77b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:26:37.309089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.308977    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-home" (OuterVolumeSpecName: "home") pod "4414e41e-4360-4b99-81a4-dbe91c8c77b3" (UID: "4414e41e-4360-4b99-81a4-dbe91c8c77b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:26:37.309089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.309020    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tls-certs\") pod \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\" (UID: \"4414e41e-4360-4b99-81a4-dbe91c8c77b3\") "
Sep 09 14:26:37.309351 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.309210    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "4414e41e-4360-4b99-81a4-dbe91c8c77b3" (UID: "4414e41e-4360-4b99-81a4-dbe91c8c77b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:26:37.309456 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.309436    2532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tmp-dir\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.309520 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.309461    2532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-home\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.309520 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.309475    2532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-model-cache\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.310932 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.310902    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-dshm" (OuterVolumeSpecName: "dshm") pod "4414e41e-4360-4b99-81a4-dbe91c8c77b3" (UID: "4414e41e-4360-4b99-81a4-dbe91c8c77b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:26:37.311043 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.310963    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4414e41e-4360-4b99-81a4-dbe91c8c77b3" (UID: "4414e41e-4360-4b99-81a4-dbe91c8c77b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:26:37.311126 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.311049    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4414e41e-4360-4b99-81a4-dbe91c8c77b3-kube-api-access-c2sw4" (OuterVolumeSpecName: "kube-api-access-c2sw4") pod "4414e41e-4360-4b99-81a4-dbe91c8c77b3" (UID: "4414e41e-4360-4b99-81a4-dbe91c8c77b3"). InnerVolumeSpecName "kube-api-access-c2sw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:26:37.312264 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.312250    2532 scope.go:117] "RemoveContainer" containerID="293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc"
Sep 09 14:26:37.312535 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:26:37.312514    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc\": container with ID starting with 293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc not found: ID does not exist" containerID="293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc"
Sep 09 14:26:37.312600 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.312540    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc"} err="failed to get container status \"293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc\": rpc error: code = NotFound desc = could not find container \"293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc\": container with ID starting with 293e8ff13e811910dc1280a6e9ee58fd6062e8e8fc3805ec0a37474cd167d1cc not found: ID does not exist"
Sep 09 14:26:37.410222 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.410198    2532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4414e41e-4360-4b99-81a4-dbe91c8c77b3-dshm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.410222 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.410218    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2sw4\" (UniqueName: \"kubernetes.io/projected/4414e41e-4360-4b99-81a4-dbe91c8c77b3-kube-api-access-c2sw4\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.410356 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.410228    2532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4414e41e-4360-4b99-81a4-dbe91c8c77b3-tls-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.514118 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.514065    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c" podUID="530702a5-679b-4479-9535-d67731cb227a" containerName="main" containerID="cri-o://d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536" gracePeriod=45
Sep 09 14:26:37.625365 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.625347    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"]
Sep 09 14:26:37.627264 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.627242    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-5cac3324-kserve-787df77ffc-6dgkj"]
Sep 09 14:26:37.750777 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.750753    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:26:37.813226 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813199    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct9tm\" (UniqueName: \"kubernetes.io/projected/530702a5-679b-4479-9535-d67731cb227a-kube-api-access-ct9tm\") pod \"530702a5-679b-4479-9535-d67731cb227a\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") "
Sep 09 14:26:37.813362 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813253    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-tmp-dir\") pod \"530702a5-679b-4479-9535-d67731cb227a\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") "
Sep 09 14:26:37.813362 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813295    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/530702a5-679b-4479-9535-d67731cb227a-tls-certs\") pod \"530702a5-679b-4479-9535-d67731cb227a\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") "
Sep 09 14:26:37.813362 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813339    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-home\") pod \"530702a5-679b-4479-9535-d67731cb227a\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") "
Sep 09 14:26:37.813502 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813366    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-dshm\") pod \"530702a5-679b-4479-9535-d67731cb227a\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") "
Sep 09 14:26:37.813502 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813411    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-model-cache\") pod \"530702a5-679b-4479-9535-d67731cb227a\" (UID: \"530702a5-679b-4479-9535-d67731cb227a\") "
Sep 09 14:26:37.813610 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813524    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "530702a5-679b-4479-9535-d67731cb227a" (UID: "530702a5-679b-4479-9535-d67731cb227a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:26:37.813716 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813696    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-home" (OuterVolumeSpecName: "home") pod "530702a5-679b-4479-9535-d67731cb227a" (UID: "530702a5-679b-4479-9535-d67731cb227a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:26:37.813716 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813705    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-model-cache" (OuterVolumeSpecName: "model-cache") pod "530702a5-679b-4479-9535-d67731cb227a" (UID: "530702a5-679b-4479-9535-d67731cb227a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:26:37.813859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813804    2532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-model-cache\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.813859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813824    2532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-tmp-dir\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.813859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.813837    2532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-home\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.815311 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.815282    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/530702a5-679b-4479-9535-d67731cb227a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "530702a5-679b-4479-9535-d67731cb227a" (UID: "530702a5-679b-4479-9535-d67731cb227a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:26:37.815417 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.815311    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530702a5-679b-4479-9535-d67731cb227a-kube-api-access-ct9tm" (OuterVolumeSpecName: "kube-api-access-ct9tm") pod "530702a5-679b-4479-9535-d67731cb227a" (UID: "530702a5-679b-4479-9535-d67731cb227a"). InnerVolumeSpecName "kube-api-access-ct9tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:26:37.815417 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.815359    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-dshm" (OuterVolumeSpecName: "dshm") pod "530702a5-679b-4479-9535-d67731cb227a" (UID: "530702a5-679b-4479-9535-d67731cb227a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:26:37.915030 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.914949    2532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/530702a5-679b-4479-9535-d67731cb227a-dshm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.915030 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.914981    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct9tm\" (UniqueName: \"kubernetes.io/projected/530702a5-679b-4479-9535-d67731cb227a-kube-api-access-ct9tm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:37.915030 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:37.915009    2532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/530702a5-679b-4479-9535-d67731cb227a-tls-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:26:38.309668 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.309639    2532 generic.go:356] "Generic (PLEG): container finished" podID="530702a5-679b-4479-9535-d67731cb227a" containerID="d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536" exitCode=0
Sep 09 14:26:38.309856 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.309679    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"
Sep 09 14:26:38.309856 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.309730    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c" event={"ID":"530702a5-679b-4479-9535-d67731cb227a","Type":"ContainerDied","Data":"d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536"}
Sep 09 14:26:38.309856 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.309778    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c" event={"ID":"530702a5-679b-4479-9535-d67731cb227a","Type":"ContainerDied","Data":"4c6b402c08074a91e1585a41b89dcc32fc07338cd776e1a37868feb777b4a310"}
Sep 09 14:26:38.309856 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.309800    2532 scope.go:117] "RemoveContainer" containerID="d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536"
Sep 09 14:26:38.319986 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.319732    2532 scope.go:117] "RemoveContainer" containerID="d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536"
Sep 09 14:26:38.320301 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:26:38.320269    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536\": container with ID starting with d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536 not found: ID does not exist" containerID="d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536"
Sep 09 14:26:38.320417 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.320313    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536"} err="failed to get container status \"d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536\": rpc error: code = NotFound desc = could not find container \"d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536\": container with ID starting with d78bc6a977bb356b1bc7fa085cd0055b34a145f3286c8fa93509c2ba8f1f2536 not found: ID does not exist"
Sep 09 14:26:38.336608 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.336585    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"]
Sep 09 14:26:38.340484 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.340461    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-5cac3324-kserve-6749769855-gmw8c"]
Sep 09 14:26:38.734891 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.734858    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4414e41e-4360-4b99-81a4-dbe91c8c77b3" path="/var/lib/kubelet/pods/4414e41e-4360-4b99-81a4-dbe91c8c77b3/volumes"
Sep 09 14:26:38.735276 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:38.735261    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="530702a5-679b-4479-9535-d67731cb227a" path="/var/lib/kubelet/pods/530702a5-679b-4479-9535-d67731cb227a/volumes"
Sep 09 14:26:48.019695 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.019661    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"]
Sep 09 14:26:48.020179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020087    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13b1e077-f1a0-4e1a-a665-c0a0b41c1861" containerName="payload-processing"
Sep 09 14:26:48.020179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020120    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="13b1e077-f1a0-4e1a-a665-c0a0b41c1861" containerName="payload-processing"
Sep 09 14:26:48.020179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020148    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4414e41e-4360-4b99-81a4-dbe91c8c77b3" containerName="main"
Sep 09 14:26:48.020179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020154    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="4414e41e-4360-4b99-81a4-dbe91c8c77b3" containerName="main"
Sep 09 14:26:48.020179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020161    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="530702a5-679b-4479-9535-d67731cb227a" containerName="main"
Sep 09 14:26:48.020179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020167    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="530702a5-679b-4479-9535-d67731cb227a" containerName="main"
Sep 09 14:26:48.020179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020173    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0ab440-e463-4359-8e8a-2264ecfccc3e" containerName="maas-api"
Sep 09 14:26:48.020179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020181    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0ab440-e463-4359-8e8a-2264ecfccc3e" containerName="maas-api"
Sep 09 14:26:48.020608 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020273    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="13b1e077-f1a0-4e1a-a665-c0a0b41c1861" containerName="payload-processing"
Sep 09 14:26:48.020608 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020289    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="530702a5-679b-4479-9535-d67731cb227a" containerName="main"
Sep 09 14:26:48.020608 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020301    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="4414e41e-4360-4b99-81a4-dbe91c8c77b3" containerName="main"
Sep 09 14:26:48.020608 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.020313    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0ab440-e463-4359-8e8a-2264ecfccc3e" containerName="maas-api"
Sep 09 14:26:48.024604 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.024585    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.027302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.027280    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-0da694-serving-cert\""
Sep 09 14:26:48.033969 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.033940    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"]
Sep 09 14:26:48.105742 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.105714    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-metrics-certs\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.105861 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.105746    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42a1f32-094c-496c-8d24-878cb79ce33f-openshift-service-ca\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.105861 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.105769    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcs7\" (UniqueName: \"kubernetes.io/projected/e42a1f32-094c-496c-8d24-878cb79ce33f-kube-api-access-jdcs7\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.105861 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.105816    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-maas-api-tls\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.206908 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.206880    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-metrics-certs\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.206908 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.206910    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42a1f32-094c-496c-8d24-878cb79ce33f-openshift-service-ca\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.207069 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.206930    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcs7\" (UniqueName: \"kubernetes.io/projected/e42a1f32-094c-496c-8d24-878cb79ce33f-kube-api-access-jdcs7\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.207069 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.206953    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-maas-api-tls\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.207507 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.207479    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42a1f32-094c-496c-8d24-878cb79ce33f-openshift-service-ca\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.209324 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.209296    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-maas-api-tls\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.209419 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.209358    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-metrics-certs\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.215382 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.215361    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcs7\" (UniqueName: \"kubernetes.io/projected/e42a1f32-094c-496c-8d24-878cb79ce33f-kube-api-access-jdcs7\") pod \"maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.337057 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.337000    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:48.609403 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.609371    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"]
Sep 09 14:26:48.610624 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:26:48.610582    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42a1f32_094c_496c_8d24_878cb79ce33f.slice/crio-01d23ac5f29a68b38348ca1aa3f5584a8b487bca816d58ea69ccd4d719dccada WatchSource:0}: Error finding container 01d23ac5f29a68b38348ca1aa3f5584a8b487bca816d58ea69ccd4d719dccada: Status 404 returned error can't find the container with id 01d23ac5f29a68b38348ca1aa3f5584a8b487bca816d58ea69ccd4d719dccada
Sep 09 14:26:48.612015 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.611996    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 14:26:48.994483 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.994450    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"]
Sep 09 14:26:48.999339 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:48.999319    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:26:49.001916 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.001897    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0da694\""
Sep 09 14:26:49.002042 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.001897    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0da694-dockercfg-6r2lx\""
Sep 09 14:26:49.008653 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.008628    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"]
Sep 09 14:26:49.116083 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.116054    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx\" (UID: \"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:26:49.116396 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.116211    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9sxn\" (UniqueName: \"kubernetes.io/projected/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-kube-api-access-s9sxn\") pod \"payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx\" (UID: \"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:26:49.216879 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.216842    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9sxn\" (UniqueName: \"kubernetes.io/projected/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-kube-api-access-s9sxn\") pod \"payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx\" (UID: \"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:26:49.217013 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.216893    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx\" (UID: \"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:26:49.217524 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.217505    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx\" (UID: \"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:26:49.224774 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.224753    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9sxn\" (UniqueName: \"kubernetes.io/projected/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-kube-api-access-s9sxn\") pod \"payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx\" (UID: \"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:26:49.311668 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.311605    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:26:49.379817 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.379773    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p" event={"ID":"e42a1f32-094c-496c-8d24-878cb79ce33f","Type":"ContainerStarted","Data":"01d23ac5f29a68b38348ca1aa3f5584a8b487bca816d58ea69ccd4d719dccada"}
Sep 09 14:26:49.582997 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:49.582962    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"]
Sep 09 14:26:49.584502 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:26:49.584475    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3c1ae4_5a93_47d3_b5bb_aff581d8c4a3.slice/crio-c3af524d46edec8134be7fe06f83d96d3da811d6f64fd5eba7df130ce2ddf2da WatchSource:0}: Error finding container c3af524d46edec8134be7fe06f83d96d3da811d6f64fd5eba7df130ce2ddf2da: Status 404 returned error can't find the container with id c3af524d46edec8134be7fe06f83d96d3da811d6f64fd5eba7df130ce2ddf2da
Sep 09 14:26:50.387257 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:50.387214    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx" event={"ID":"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3","Type":"ContainerStarted","Data":"1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610"}
Sep 09 14:26:50.387680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:50.387279    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx" event={"ID":"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3","Type":"ContainerStarted","Data":"c3af524d46edec8134be7fe06f83d96d3da811d6f64fd5eba7df130ce2ddf2da"}
Sep 09 14:26:50.387680 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:50.387329    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:26:50.426267 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:50.426223    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx" podStartSLOduration=2.4262088520000002 podStartE2EDuration="2.426208852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:26:50.42424481 +0000 UTC m=+944.187040050" watchObservedRunningTime="2026-09-09 14:26:50.426208852 +0000 UTC m=+944.189004091"
Sep 09 14:26:51.394650 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:51.394614    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p" event={"ID":"e42a1f32-094c-496c-8d24-878cb79ce33f","Type":"ContainerStarted","Data":"9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059"}
Sep 09 14:26:51.395124 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:51.394908    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:26:51.415519 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:51.415475    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p" podStartSLOduration=1.55382906 podStartE2EDuration="3.415460629s" totalImagesPullingTime="1.861631569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:26:51.413849 +0000 UTC m=+945.176644243" watchObservedRunningTime="2026-09-09 14:26:51.415460629 +0000 UTC m=+945.178255868"
Sep 09 14:26:57.405772 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:26:57.405742    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:27:01.397273 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:01.397244    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:27:16.493832 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.493794    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"]
Sep 09 14:27:16.497782 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.497761    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.501666 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.501642    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d85f87\"/\"test-model-d85f87-kserve-self-signed-certs\""
Sep 09 14:27:16.501959 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.501939    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d85f87\"/\"kube-root-ca.crt\""
Sep 09 14:27:16.502087 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.502068    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d85f87\"/\"openshift-service-ca.crt\""
Sep 09 14:27:16.502189 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.502131    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d85f87\"/\"default-dockercfg-fsf6j\""
Sep 09 14:27:16.507522 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.507501    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"]
Sep 09 14:27:16.553905 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.553879    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tmp-dir\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.554047 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.553917    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9dtv\" (UniqueName: \"kubernetes.io/projected/4a3aa9c8-ea76-43dc-b31a-78714701aba9-kube-api-access-z9dtv\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.554047 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.554009    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-model-cache\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.554195 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.554050    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-home\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.554195 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.554080    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-dshm\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.554195 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.554161    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tls-certs\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.655363 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.655337    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-model-cache\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.655363 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.655365    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-home\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.655516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.655383    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-dshm\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.655516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.655405    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tls-certs\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.655516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.655461    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tmp-dir\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.655516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.655502    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9dtv\" (UniqueName: \"kubernetes.io/projected/4a3aa9c8-ea76-43dc-b31a-78714701aba9-kube-api-access-z9dtv\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.655787 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.655765    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-model-cache\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.655841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.655795    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-home\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.655883 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.655873    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tmp-dir\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.657657 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.657630    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-dshm\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.657937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.657918    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tls-certs\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.676937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.676902    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9dtv\" (UniqueName: \"kubernetes.io/projected/4a3aa9c8-ea76-43dc-b31a-78714701aba9-kube-api-access-z9dtv\") pod \"test-model-d85f87-kserve-7bd6ffb469-4w7d5\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") " pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:16.807717 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:16.807646    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:17.077722 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:17.077694    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"]
Sep 09 14:27:17.079305 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:27:17.079278    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a3aa9c8_ea76_43dc_b31a_78714701aba9.slice/crio-ac1688427bfd88ad330f3803fe9993a8cbb43b93f2f108e78e7d4c4ce7cdf7f6 WatchSource:0}: Error finding container ac1688427bfd88ad330f3803fe9993a8cbb43b93f2f108e78e7d4c4ce7cdf7f6: Status 404 returned error can't find the container with id ac1688427bfd88ad330f3803fe9993a8cbb43b93f2f108e78e7d4c4ce7cdf7f6
Sep 09 14:27:17.531552 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:17.531507    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5" event={"ID":"4a3aa9c8-ea76-43dc-b31a-78714701aba9","Type":"ContainerStarted","Data":"76fa2bc0329addb006322f45b29d78f08832247a86651cae4704e353e8bd915f"}
Sep 09 14:27:17.531552 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:17.531553    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5" event={"ID":"4a3aa9c8-ea76-43dc-b31a-78714701aba9","Type":"ContainerStarted","Data":"ac1688427bfd88ad330f3803fe9993a8cbb43b93f2f108e78e7d4c4ce7cdf7f6"}
Sep 09 14:27:17.548046 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:17.547993    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5" podStartSLOduration=1.547974559 podStartE2EDuration="1.547974559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:27:17.547373305 +0000 UTC m=+971.310168546" watchObservedRunningTime="2026-09-09 14:27:17.547974559 +0000 UTC m=+971.310769800"
Sep 09 14:27:26.808382 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:26.808348    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:26.808837 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:26.808393    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:26.820499 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:26.820473    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:27.593387 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:27.593355    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:27:45.339344 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.339268    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"]
Sep 09 14:27:45.339885 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.339502    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p" podUID="e42a1f32-094c-496c-8d24-878cb79ce33f" containerName="maas-api" containerID="cri-o://9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059" gracePeriod=30
Sep 09 14:27:45.403002 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.402964    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"]
Sep 09 14:27:45.403234 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.403194    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx" podUID="0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3" containerName="payload-pre-processing" containerID="cri-o://1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610" gracePeriod=30
Sep 09 14:27:45.650275 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.650250    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:27:45.678232 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.678163    2532 generic.go:356] "Generic (PLEG): container finished" podID="e42a1f32-094c-496c-8d24-878cb79ce33f" containerID="9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059" exitCode=0
Sep 09 14:27:45.678402 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.678180    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"
Sep 09 14:27:45.678526 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.678198    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p" event={"ID":"e42a1f32-094c-496c-8d24-878cb79ce33f","Type":"ContainerDied","Data":"9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059"}
Sep 09 14:27:45.678646 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.678563    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p" event={"ID":"e42a1f32-094c-496c-8d24-878cb79ce33f","Type":"ContainerDied","Data":"01d23ac5f29a68b38348ca1aa3f5584a8b487bca816d58ea69ccd4d719dccada"}
Sep 09 14:27:45.678646 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.678604    2532 scope.go:117] "RemoveContainer" containerID="9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059"
Sep 09 14:27:45.679293 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.679255    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:27:45.680977 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.680954    2532 generic.go:356] "Generic (PLEG): container finished" podID="0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3" containerID="1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610" exitCode=0
Sep 09 14:27:45.681087 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.681064    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx" event={"ID":"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3","Type":"ContainerDied","Data":"1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610"}
Sep 09 14:27:45.681190 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.681147    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx" event={"ID":"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3","Type":"ContainerDied","Data":"c3af524d46edec8134be7fe06f83d96d3da811d6f64fd5eba7df130ce2ddf2da"}
Sep 09 14:27:45.689009 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.688994    2532 scope.go:117] "RemoveContainer" containerID="9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059"
Sep 09 14:27:45.689315 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:27:45.689288    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059\": container with ID starting with 9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059 not found: ID does not exist" containerID="9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059"
Sep 09 14:27:45.689398 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.689326    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059"} err="failed to get container status \"9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059\": rpc error: code = NotFound desc = could not find container \"9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059\": container with ID starting with 9c6fae19b8624954dbe111b95ebcce4ca4611c0e9d44f866532e0e59b0d19059 not found: ID does not exist"
Sep 09 14:27:45.689398 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.689347    2532 scope.go:117] "RemoveContainer" containerID="1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610"
Sep 09 14:27:45.696835 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.696818    2532 scope.go:117] "RemoveContainer" containerID="1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610"
Sep 09 14:27:45.697122 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:27:45.697076    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610\": container with ID starting with 1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610 not found: ID does not exist" containerID="1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610"
Sep 09 14:27:45.697189 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.697121    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610"} err="failed to get container status \"1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610\": rpc error: code = NotFound desc = could not find container \"1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610\": container with ID starting with 1576a7d951f53f60d38b9af3e59c1e8a05572d526e03ed28978dba4ae3029610 not found: ID does not exist"
Sep 09 14:27:45.810251 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.810220    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9sxn\" (UniqueName: \"kubernetes.io/projected/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-kube-api-access-s9sxn\") pod \"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3\" (UID: \"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3\") "
Sep 09 14:27:45.810438 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.810294    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42a1f32-094c-496c-8d24-878cb79ce33f-openshift-service-ca\") pod \"e42a1f32-094c-496c-8d24-878cb79ce33f\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") "
Sep 09 14:27:45.810438 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.810316    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-plugins-config-volume\") pod \"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3\" (UID: \"0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3\") "
Sep 09 14:27:45.810438 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.810340    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdcs7\" (UniqueName: \"kubernetes.io/projected/e42a1f32-094c-496c-8d24-878cb79ce33f-kube-api-access-jdcs7\") pod \"e42a1f32-094c-496c-8d24-878cb79ce33f\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") "
Sep 09 14:27:45.810438 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.810382    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-maas-api-tls\") pod \"e42a1f32-094c-496c-8d24-878cb79ce33f\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") "
Sep 09 14:27:45.810438 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.810410    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-metrics-certs\") pod \"e42a1f32-094c-496c-8d24-878cb79ce33f\" (UID: \"e42a1f32-094c-496c-8d24-878cb79ce33f\") "
Sep 09 14:27:45.810810 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.810784    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42a1f32-094c-496c-8d24-878cb79ce33f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e42a1f32-094c-496c-8d24-878cb79ce33f" (UID: "e42a1f32-094c-496c-8d24-878cb79ce33f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:27:45.810881 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.810792    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3" (UID: "0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:27:45.812448 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.812414    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e42a1f32-094c-496c-8d24-878cb79ce33f" (UID: "e42a1f32-094c-496c-8d24-878cb79ce33f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:27:45.812544 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.812459    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42a1f32-094c-496c-8d24-878cb79ce33f-kube-api-access-jdcs7" (OuterVolumeSpecName: "kube-api-access-jdcs7") pod "e42a1f32-094c-496c-8d24-878cb79ce33f" (UID: "e42a1f32-094c-496c-8d24-878cb79ce33f"). InnerVolumeSpecName "kube-api-access-jdcs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:27:45.812720 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.812704    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-kube-api-access-s9sxn" (OuterVolumeSpecName: "kube-api-access-s9sxn") pod "0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3" (UID: "0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3"). InnerVolumeSpecName "kube-api-access-s9sxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:27:45.812780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.812704    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e42a1f32-094c-496c-8d24-878cb79ce33f" (UID: "e42a1f32-094c-496c-8d24-878cb79ce33f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:27:45.912174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.912090    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42a1f32-094c-496c-8d24-878cb79ce33f-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:27:45.912174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.912140    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:27:45.912174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.912152    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdcs7\" (UniqueName: \"kubernetes.io/projected/e42a1f32-094c-496c-8d24-878cb79ce33f-kube-api-access-jdcs7\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:27:45.912174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.912161    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:27:45.912363 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.912238    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e42a1f32-094c-496c-8d24-878cb79ce33f-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:27:45.912363 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:45.912247    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9sxn\" (UniqueName: \"kubernetes.io/projected/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3-kube-api-access-s9sxn\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:27:46.001764 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:46.001733    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"]
Sep 09 14:27:46.007253 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:46.007231    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0da694-69c9ffc7db-rfk5p"]
Sep 09 14:27:46.687895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:46.687829    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"
Sep 09 14:27:46.707734 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:46.707704    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"]
Sep 09 14:27:46.711724 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:46.711700    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0da694-5b56b94d66-dbjnx"]
Sep 09 14:27:46.734920 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:46.734896    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3" path="/var/lib/kubelet/pods/0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3/volumes"
Sep 09 14:27:46.735309 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:27:46.735295    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e42a1f32-094c-496c-8d24-878cb79ce33f" path="/var/lib/kubelet/pods/e42a1f32-094c-496c-8d24-878cb79ce33f/volumes"
Sep 09 14:28:01.102301 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.102223    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"]
Sep 09 14:28:01.102913 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.102892    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3" containerName="payload-pre-processing"
Sep 09 14:28:01.102959 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.102917    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3" containerName="payload-pre-processing"
Sep 09 14:28:01.102959 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.102931    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e42a1f32-094c-496c-8d24-878cb79ce33f" containerName="maas-api"
Sep 09 14:28:01.102959 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.102941    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="e42a1f32-094c-496c-8d24-878cb79ce33f" containerName="maas-api"
Sep 09 14:28:01.103055 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.103048    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3c1ae4-5a93-47d3-b5bb-aff581d8c4a3" containerName="payload-pre-processing"
Sep 09 14:28:01.103089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.103063    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="e42a1f32-094c-496c-8d24-878cb79ce33f" containerName="maas-api"
Sep 09 14:28:01.108015 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.107997    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.111017 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.110876    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-aa5470\"/\"noten-model-aa5470-kserve-self-signed-certs\""
Sep 09 14:28:01.111017 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.110945    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-aa5470\"/\"kube-root-ca.crt\""
Sep 09 14:28:01.111282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.111125    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-aa5470\"/\"default-dockercfg-lxcmt\""
Sep 09 14:28:01.111282 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.111253    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-aa5470\"/\"openshift-service-ca.crt\""
Sep 09 14:28:01.112756 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.112734    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"]
Sep 09 14:28:01.133542 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.133517    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tls-certs\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.133665 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.133584    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-dshm\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.133665 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.133613    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tmp-dir\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.133761 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.133674    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvpjl\" (UniqueName: \"kubernetes.io/projected/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-kube-api-access-dvpjl\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.133802 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.133780    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-model-cache\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.133843 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.133808    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-home\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.235250 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.235229    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-dshm\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.235351 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.235258    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tmp-dir\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.235351 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.235294    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvpjl\" (UniqueName: \"kubernetes.io/projected/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-kube-api-access-dvpjl\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.235438 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.235410    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-model-cache\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.235483 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.235436    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-home\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.235535 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.235483    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tls-certs\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.235684 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.235655    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tmp-dir\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.235806 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.235734    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-model-cache\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.235856 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.235806    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-home\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.237335 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.237318    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-dshm\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.237771 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.237756    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tls-certs\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.242859 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.242840    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvpjl\" (UniqueName: \"kubernetes.io/projected/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-kube-api-access-dvpjl\") pod \"noten-model-aa5470-kserve-765c89d94c-kxr6m\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") " pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.419322 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.419239    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:01.691160 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.691138    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"]
Sep 09 14:28:01.693066 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:28:01.693043    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ec8ea3_fb1b_4503_9093_279a6dbf462c.slice/crio-d9dc43f111bafade8f6e26ba2204988a9e6cf64b22b738ebe84c82ecebfb7e00 WatchSource:0}: Error finding container d9dc43f111bafade8f6e26ba2204988a9e6cf64b22b738ebe84c82ecebfb7e00: Status 404 returned error can't find the container with id d9dc43f111bafade8f6e26ba2204988a9e6cf64b22b738ebe84c82ecebfb7e00
Sep 09 14:28:01.760813 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.760789    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m" event={"ID":"c2ec8ea3-fb1b-4503-9093-279a6dbf462c","Type":"ContainerStarted","Data":"3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343"}
Sep 09 14:28:01.760915 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.760829    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m" event={"ID":"c2ec8ea3-fb1b-4503-9093-279a6dbf462c","Type":"ContainerStarted","Data":"d9dc43f111bafade8f6e26ba2204988a9e6cf64b22b738ebe84c82ecebfb7e00"}
Sep 09 14:28:01.779512 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:01.779470    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m" podStartSLOduration=0.779454354 podStartE2EDuration="779.454354ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:28:01.778332048 +0000 UTC m=+1015.541127287" watchObservedRunningTime="2026-09-09 14:28:01.779454354 +0000 UTC m=+1015.542249655"
Sep 09 14:28:11.420373 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:11.420339    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:11.420792 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:11.420409    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:11.432517 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:11.432495    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:11.823204 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:11.823178    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:14.399983 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.399943    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"]
Sep 09 14:28:14.405048 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.405013    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.407766 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.407732    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d94c06b6-serving-cert\""
Sep 09 14:28:14.416436 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.416413    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"]
Sep 09 14:28:14.556265 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.556238    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2daa202-2a81-45a7-9827-80215b8ed258-openshift-service-ca\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.556395 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.556296    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-metrics-certs\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.556395 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.556356    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gcf7\" (UniqueName: \"kubernetes.io/projected/c2daa202-2a81-45a7-9827-80215b8ed258-kube-api-access-8gcf7\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.556395 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.556382    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-maas-api-tls\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.657667 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.657608    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2daa202-2a81-45a7-9827-80215b8ed258-openshift-service-ca\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.657758 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.657670    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-metrics-certs\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.657758 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.657709    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gcf7\" (UniqueName: \"kubernetes.io/projected/c2daa202-2a81-45a7-9827-80215b8ed258-kube-api-access-8gcf7\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.657758 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.657734    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-maas-api-tls\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.658349 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.658328    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2daa202-2a81-45a7-9827-80215b8ed258-openshift-service-ca\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.660065 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.660037    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-maas-api-tls\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.660167 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.660037    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-metrics-certs\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.665717 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.665693    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gcf7\" (UniqueName: \"kubernetes.io/projected/c2daa202-2a81-45a7-9827-80215b8ed258-kube-api-access-8gcf7\") pod \"maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.716703 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.716681    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:14.991775 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:14.991740    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"]
Sep 09 14:28:14.993733 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:28:14.993706    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2daa202_2a81_45a7_9827_80215b8ed258.slice/crio-d245a9a114dd29a86ca8e7c3978bc1e62e87fcfa8106c574ffb64fc882ece39e WatchSource:0}: Error finding container d245a9a114dd29a86ca8e7c3978bc1e62e87fcfa8106c574ffb64fc882ece39e: Status 404 returned error can't find the container with id d245a9a114dd29a86ca8e7c3978bc1e62e87fcfa8106c574ffb64fc882ece39e
Sep 09 14:28:15.054681 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.054655    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"]
Sep 09 14:28:15.059359 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.059340    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:15.062360 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.062341    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d94c06b6-dockercfg-4sq5w\""
Sep 09 14:28:15.062573 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.062556    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d94c06b6\""
Sep 09 14:28:15.068708 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.068687    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"]
Sep 09 14:28:15.163507 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.163475    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973e3530-023a-4161-bda6-b54d52563c06-plugins-config-volume\") pod \"payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4\" (UID: \"973e3530-023a-4161-bda6-b54d52563c06\") " pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:15.163641 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.163549    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7sv\" (UniqueName: \"kubernetes.io/projected/973e3530-023a-4161-bda6-b54d52563c06-kube-api-access-xz7sv\") pod \"payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4\" (UID: \"973e3530-023a-4161-bda6-b54d52563c06\") " pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:15.264491 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.264420    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7sv\" (UniqueName: \"kubernetes.io/projected/973e3530-023a-4161-bda6-b54d52563c06-kube-api-access-xz7sv\") pod \"payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4\" (UID: \"973e3530-023a-4161-bda6-b54d52563c06\") " pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:15.264627 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.264551    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973e3530-023a-4161-bda6-b54d52563c06-plugins-config-volume\") pod \"payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4\" (UID: \"973e3530-023a-4161-bda6-b54d52563c06\") " pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:15.265269 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.265249    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973e3530-023a-4161-bda6-b54d52563c06-plugins-config-volume\") pod \"payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4\" (UID: \"973e3530-023a-4161-bda6-b54d52563c06\") " pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:15.276658 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.276635    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7sv\" (UniqueName: \"kubernetes.io/projected/973e3530-023a-4161-bda6-b54d52563c06-kube-api-access-xz7sv\") pod \"payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4\" (UID: \"973e3530-023a-4161-bda6-b54d52563c06\") " pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:15.277565 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.277546    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"]
Sep 09 14:28:15.371789 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.371758    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:15.649157 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.649131    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"]
Sep 09 14:28:15.651603 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:28:15.651564    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973e3530_023a_4161_bda6_b54d52563c06.slice/crio-3ea7afb3335fdc3ba9238d073c2396eac37b94b69a0b6bb8d755c3fb1e12f534 WatchSource:0}: Error finding container 3ea7afb3335fdc3ba9238d073c2396eac37b94b69a0b6bb8d755c3fb1e12f534: Status 404 returned error can't find the container with id 3ea7afb3335fdc3ba9238d073c2396eac37b94b69a0b6bb8d755c3fb1e12f534
Sep 09 14:28:15.841633 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.841592    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4" event={"ID":"973e3530-023a-4161-bda6-b54d52563c06","Type":"ContainerStarted","Data":"fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104"}
Sep 09 14:28:15.841828 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.841673    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4" event={"ID":"973e3530-023a-4161-bda6-b54d52563c06","Type":"ContainerStarted","Data":"3ea7afb3335fdc3ba9238d073c2396eac37b94b69a0b6bb8d755c3fb1e12f534"}
Sep 09 14:28:15.841828 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.841716    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:15.843148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.843122    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b" event={"ID":"c2daa202-2a81-45a7-9827-80215b8ed258","Type":"ContainerStarted","Data":"d245a9a114dd29a86ca8e7c3978bc1e62e87fcfa8106c574ffb64fc882ece39e"}
Sep 09 14:28:15.859488 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:15.859447    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4" podStartSLOduration=0.859434706 podStartE2EDuration="859.434706ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:28:15.857566262 +0000 UTC m=+1029.620361503" watchObservedRunningTime="2026-09-09 14:28:15.859434706 +0000 UTC m=+1029.622229945"
Sep 09 14:28:17.854758 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:17.854723    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b" event={"ID":"c2daa202-2a81-45a7-9827-80215b8ed258","Type":"ContainerStarted","Data":"fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19"}
Sep 09 14:28:17.855138 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:17.854793    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:17.873981 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:17.873927    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b" podStartSLOduration=1.810173232 podStartE2EDuration="3.873912587s" totalImagesPullingTime="2.063739355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:28:17.871029672 +0000 UTC m=+1031.633824911" watchObservedRunningTime="2026-09-09 14:28:17.873912587 +0000 UTC m=+1031.636707827"
Sep 09 14:28:23.865413 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:23.865384    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:26.850720 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:26.850686    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:28.828657 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:28.828622    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"]
Sep 09 14:28:30.296962 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.296929    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m" podUID="c2ec8ea3-fb1b-4503-9093-279a6dbf462c" containerName="main" containerID="cri-o://3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343" gracePeriod=45
Sep 09 14:28:30.543289 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.543268    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:30.709192 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709106    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tls-certs\") pod \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") "
Sep 09 14:28:30.709192 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709163    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-dshm\") pod \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") "
Sep 09 14:28:30.709192 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709187    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-home\") pod \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") "
Sep 09 14:28:30.709441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709265    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-model-cache\") pod \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") "
Sep 09 14:28:30.709441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709323    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvpjl\" (UniqueName: \"kubernetes.io/projected/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-kube-api-access-dvpjl\") pod \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") "
Sep 09 14:28:30.709441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709346    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tmp-dir\") pod \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\" (UID: \"c2ec8ea3-fb1b-4503-9093-279a6dbf462c\") "
Sep 09 14:28:30.709441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709417    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-home" (OuterVolumeSpecName: "home") pod "c2ec8ea3-fb1b-4503-9093-279a6dbf462c" (UID: "c2ec8ea3-fb1b-4503-9093-279a6dbf462c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:28:30.709610 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709521    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2ec8ea3-fb1b-4503-9093-279a6dbf462c" (UID: "c2ec8ea3-fb1b-4503-9093-279a6dbf462c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:28:30.709610 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709599    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2ec8ea3-fb1b-4503-9093-279a6dbf462c" (UID: "c2ec8ea3-fb1b-4503-9093-279a6dbf462c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:28:30.709770 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709752    2532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-home\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:30.709822 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709777    2532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-model-cache\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:30.709822 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.709794    2532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tmp-dir\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:30.711277 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.711253    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-dshm" (OuterVolumeSpecName: "dshm") pod "c2ec8ea3-fb1b-4503-9093-279a6dbf462c" (UID: "c2ec8ea3-fb1b-4503-9093-279a6dbf462c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:28:30.711375 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.711304    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2ec8ea3-fb1b-4503-9093-279a6dbf462c" (UID: "c2ec8ea3-fb1b-4503-9093-279a6dbf462c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:28:30.711375 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.711352    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-kube-api-access-dvpjl" (OuterVolumeSpecName: "kube-api-access-dvpjl") pod "c2ec8ea3-fb1b-4503-9093-279a6dbf462c" (UID: "c2ec8ea3-fb1b-4503-9093-279a6dbf462c"). InnerVolumeSpecName "kube-api-access-dvpjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:28:30.810686 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.810660    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvpjl\" (UniqueName: \"kubernetes.io/projected/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-kube-api-access-dvpjl\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:30.810686 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.810686    2532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-tls-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:30.810822 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.810695    2532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec8ea3-fb1b-4503-9093-279a6dbf462c-dshm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:30.923060 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.923028    2532 generic.go:356] "Generic (PLEG): container finished" podID="c2ec8ea3-fb1b-4503-9093-279a6dbf462c" containerID="3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343" exitCode=0
Sep 09 14:28:30.923216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.923082    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m" event={"ID":"c2ec8ea3-fb1b-4503-9093-279a6dbf462c","Type":"ContainerDied","Data":"3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343"}
Sep 09 14:28:30.923216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.923090    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"
Sep 09 14:28:30.923216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.923146    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m" event={"ID":"c2ec8ea3-fb1b-4503-9093-279a6dbf462c","Type":"ContainerDied","Data":"d9dc43f111bafade8f6e26ba2204988a9e6cf64b22b738ebe84c82ecebfb7e00"}
Sep 09 14:28:30.923216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.923166    2532 scope.go:117] "RemoveContainer" containerID="3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343"
Sep 09 14:28:30.932033 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.932014    2532 scope.go:117] "RemoveContainer" containerID="3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343"
Sep 09 14:28:30.932309 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:28:30.932288    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343\": container with ID starting with 3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343 not found: ID does not exist" containerID="3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343"
Sep 09 14:28:30.932374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.932316    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343"} err="failed to get container status \"3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343\": rpc error: code = NotFound desc = could not find container \"3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343\": container with ID starting with 3046e3d5ae90b062a81a976aca163f76b1b6b3b5d0b69f80edaf18820088a343 not found: ID does not exist"
Sep 09 14:28:30.940519 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.940497    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"]
Sep 09 14:28:30.946333 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:30.946313    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-aa5470/noten-model-aa5470-kserve-765c89d94c-kxr6m"]
Sep 09 14:28:32.734628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:32.734594    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ec8ea3-fb1b-4503-9093-279a6dbf462c" path="/var/lib/kubelet/pods/c2ec8ea3-fb1b-4503-9093-279a6dbf462c/volumes"
Sep 09 14:28:39.731656 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:39.731624    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"]
Sep 09 14:28:39.732182 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:39.731792    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf" podUID="6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9" containerName="maas-api" containerID="cri-o://1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946" gracePeriod=30
Sep 09 14:28:39.792256 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:39.792226    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"]
Sep 09 14:28:39.792467 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:39.792421    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r" podUID="95a390b7-6abc-440b-9513-5e06d611abd7" containerName="payload-pre-processing" containerID="cri-o://510f36dcd65aa665998e5120b50005bfb0d22b12d89af2537fc369a4c122e54c" gracePeriod=30
Sep 09 14:28:39.972437 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:39.972407    2532 generic.go:356] "Generic (PLEG): container finished" podID="95a390b7-6abc-440b-9513-5e06d611abd7" containerID="510f36dcd65aa665998e5120b50005bfb0d22b12d89af2537fc369a4c122e54c" exitCode=0
Sep 09 14:28:39.972560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:39.972475    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r" event={"ID":"95a390b7-6abc-440b-9513-5e06d611abd7","Type":"ContainerDied","Data":"510f36dcd65aa665998e5120b50005bfb0d22b12d89af2537fc369a4c122e54c"}
Sep 09 14:28:40.093070 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.093044    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:28:40.169841 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.169820    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:28:40.191813 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.191792    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-maas-api-tls\") pod \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") "
Sep 09 14:28:40.191946 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.191831    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-openshift-service-ca\") pod \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") "
Sep 09 14:28:40.191946 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.191912    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-metrics-certs\") pod \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") "
Sep 09 14:28:40.191946 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.191937    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a390b7-6abc-440b-9513-5e06d611abd7-plugins-config-volume\") pod \"95a390b7-6abc-440b-9513-5e06d611abd7\" (UID: \"95a390b7-6abc-440b-9513-5e06d611abd7\") "
Sep 09 14:28:40.192660 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.191973    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n24nl\" (UniqueName: \"kubernetes.io/projected/95a390b7-6abc-440b-9513-5e06d611abd7-kube-api-access-n24nl\") pod \"95a390b7-6abc-440b-9513-5e06d611abd7\" (UID: \"95a390b7-6abc-440b-9513-5e06d611abd7\") "
Sep 09 14:28:40.192660 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.192037    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvrd8\" (UniqueName: \"kubernetes.io/projected/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-kube-api-access-hvrd8\") pod \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\" (UID: \"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9\") "
Sep 09 14:28:40.192660 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.192579    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9" (UID: "6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:28:40.192829 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.192729    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a390b7-6abc-440b-9513-5e06d611abd7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95a390b7-6abc-440b-9513-5e06d611abd7" (UID: "95a390b7-6abc-440b-9513-5e06d611abd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:28:40.194221 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.194184    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9" (UID: "6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:28:40.194601 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.194576    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a390b7-6abc-440b-9513-5e06d611abd7-kube-api-access-n24nl" (OuterVolumeSpecName: "kube-api-access-n24nl") pod "95a390b7-6abc-440b-9513-5e06d611abd7" (UID: "95a390b7-6abc-440b-9513-5e06d611abd7"). InnerVolumeSpecName "kube-api-access-n24nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:28:40.194699 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.194602    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9" (UID: "6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:28:40.194766 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.194715    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-kube-api-access-hvrd8" (OuterVolumeSpecName: "kube-api-access-hvrd8") pod "6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9" (UID: "6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9"). InnerVolumeSpecName "kube-api-access-hvrd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:28:40.293503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.293446    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:40.293503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.293470    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a390b7-6abc-440b-9513-5e06d611abd7-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:40.293503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.293480    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n24nl\" (UniqueName: \"kubernetes.io/projected/95a390b7-6abc-440b-9513-5e06d611abd7-kube-api-access-n24nl\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:40.293503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.293488    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvrd8\" (UniqueName: \"kubernetes.io/projected/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-kube-api-access-hvrd8\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:40.293503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.293496    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:40.293503 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.293506    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:40.979296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.979266    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"
Sep 09 14:28:40.979710 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.979265    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r" event={"ID":"95a390b7-6abc-440b-9513-5e06d611abd7","Type":"ContainerDied","Data":"e171796704ff7a25fa106ce9246c005916d6173e7d7227266102595a4dd8dce5"}
Sep 09 14:28:40.979710 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.979421    2532 scope.go:117] "RemoveContainer" containerID="510f36dcd65aa665998e5120b50005bfb0d22b12d89af2537fc369a4c122e54c"
Sep 09 14:28:40.980930 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.980826    2532 generic.go:356] "Generic (PLEG): container finished" podID="6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9" containerID="1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946" exitCode=0
Sep 09 14:28:40.980930 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.980875    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"
Sep 09 14:28:40.980930 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.980909    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf" event={"ID":"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9","Type":"ContainerDied","Data":"1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946"}
Sep 09 14:28:40.981087 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.980967    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf" event={"ID":"6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9","Type":"ContainerDied","Data":"4d5f39d5482fcb86a381a1f6e0725c10e6e8a0420aa578f94f2424edefa3fc1a"}
Sep 09 14:28:40.988662 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.988646    2532 scope.go:117] "RemoveContainer" containerID="1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946"
Sep 09 14:28:40.996361 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.996344    2532 scope.go:117] "RemoveContainer" containerID="1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946"
Sep 09 14:28:40.996611 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:28:40.996589    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946\": container with ID starting with 1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946 not found: ID does not exist" containerID="1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946"
Sep 09 14:28:40.996670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.996616    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946"} err="failed to get container status \"1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946\": rpc error: code = NotFound desc = could not find container \"1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946\": container with ID starting with 1f3d2a8a17b6131c820b09d48d9437889f28cade2f534580c1e3cc57d484b946 not found: ID does not exist"
Sep 09 14:28:40.999850 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:40.999829    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"]
Sep 09 14:28:41.004356 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:41.004327    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a0fa5a-6984f49d69-5hrnf"]
Sep 09 14:28:41.016899 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:41.016878    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"]
Sep 09 14:28:41.018713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:41.018693    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a0fa5a-9c857c65d-dsr2r"]
Sep 09 14:28:42.734399 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:42.734369    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9" path="/var/lib/kubelet/pods/6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9/volumes"
Sep 09 14:28:42.734751 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:42.734705    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95a390b7-6abc-440b-9513-5e06d611abd7" path="/var/lib/kubelet/pods/95a390b7-6abc-440b-9513-5e06d611abd7/volumes"
Sep 09 14:28:43.848874 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:43.848840    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5" podUID="4a3aa9c8-ea76-43dc-b31a-78714701aba9" containerName="main" containerID="cri-o://76fa2bc0329addb006322f45b29d78f08832247a86651cae4704e353e8bd915f" gracePeriod=45
Sep 09 14:28:44.001768 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.001741    2532 generic.go:356] "Generic (PLEG): container finished" podID="4a3aa9c8-ea76-43dc-b31a-78714701aba9" containerID="76fa2bc0329addb006322f45b29d78f08832247a86651cae4704e353e8bd915f" exitCode=0
Sep 09 14:28:44.001951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.001829    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5" event={"ID":"4a3aa9c8-ea76-43dc-b31a-78714701aba9","Type":"ContainerDied","Data":"76fa2bc0329addb006322f45b29d78f08832247a86651cae4704e353e8bd915f"}
Sep 09 14:28:44.099036 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.098981    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:28:44.226323 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.226295    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tls-certs\") pod \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") "
Sep 09 14:28:44.226451 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.226331    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9dtv\" (UniqueName: \"kubernetes.io/projected/4a3aa9c8-ea76-43dc-b31a-78714701aba9-kube-api-access-z9dtv\") pod \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") "
Sep 09 14:28:44.226451 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.226372    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tmp-dir\") pod \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") "
Sep 09 14:28:44.226451 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.226393    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-dshm\") pod \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") "
Sep 09 14:28:44.226574 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.226460    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-home\") pod \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") "
Sep 09 14:28:44.226574 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.226504    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-model-cache\") pod \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\" (UID: \"4a3aa9c8-ea76-43dc-b31a-78714701aba9\") "
Sep 09 14:28:44.226660 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.226606    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a3aa9c8-ea76-43dc-b31a-78714701aba9" (UID: "4a3aa9c8-ea76-43dc-b31a-78714701aba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:28:44.226787 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.226763    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-home" (OuterVolumeSpecName: "home") pod "4a3aa9c8-ea76-43dc-b31a-78714701aba9" (UID: "4a3aa9c8-ea76-43dc-b31a-78714701aba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:28:44.226847 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.226787    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a3aa9c8-ea76-43dc-b31a-78714701aba9" (UID: "4a3aa9c8-ea76-43dc-b31a-78714701aba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:28:44.226904 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.226869    2532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tmp-dir\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:44.228588 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.228562    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-dshm" (OuterVolumeSpecName: "dshm") pod "4a3aa9c8-ea76-43dc-b31a-78714701aba9" (UID: "4a3aa9c8-ea76-43dc-b31a-78714701aba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:28:44.228697 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.228585    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a3aa9c8-ea76-43dc-b31a-78714701aba9" (UID: "4a3aa9c8-ea76-43dc-b31a-78714701aba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:28:44.228697 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.228650    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3aa9c8-ea76-43dc-b31a-78714701aba9-kube-api-access-z9dtv" (OuterVolumeSpecName: "kube-api-access-z9dtv") pod "4a3aa9c8-ea76-43dc-b31a-78714701aba9" (UID: "4a3aa9c8-ea76-43dc-b31a-78714701aba9"). InnerVolumeSpecName "kube-api-access-z9dtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:28:44.328309 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.328287    2532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-dshm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:44.328309 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.328305    2532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-home\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:44.328434 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.328316    2532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3aa9c8-ea76-43dc-b31a-78714701aba9-model-cache\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:44.328434 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.328325    2532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3aa9c8-ea76-43dc-b31a-78714701aba9-tls-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:44.328434 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:44.328333    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9dtv\" (UniqueName: \"kubernetes.io/projected/4a3aa9c8-ea76-43dc-b31a-78714701aba9-kube-api-access-z9dtv\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:45.008089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:45.008016    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"
Sep 09 14:28:45.008089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:45.008032    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5" event={"ID":"4a3aa9c8-ea76-43dc-b31a-78714701aba9","Type":"ContainerDied","Data":"ac1688427bfd88ad330f3803fe9993a8cbb43b93f2f108e78e7d4c4ce7cdf7f6"}
Sep 09 14:28:45.008089 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:45.008083    2532 scope.go:117] "RemoveContainer" containerID="76fa2bc0329addb006322f45b29d78f08832247a86651cae4704e353e8bd915f"
Sep 09 14:28:45.025153 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:45.025127    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"]
Sep 09 14:28:45.027011 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:45.026988    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d85f87/test-model-d85f87-kserve-7bd6ffb469-4w7d5"]
Sep 09 14:28:46.735340 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:46.735307    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a3aa9c8-ea76-43dc-b31a-78714701aba9" path="/var/lib/kubelet/pods/4a3aa9c8-ea76-43dc-b31a-78714701aba9/volumes"
Sep 09 14:28:56.434466 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.434431    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"]
Sep 09 14:28:56.434842 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.434678    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b" podUID="c2daa202-2a81-45a7-9827-80215b8ed258" containerName="maas-api" containerID="cri-o://fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19" gracePeriod=30
Sep 09 14:28:56.494930 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.494897    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"]
Sep 09 14:28:56.495148 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.495116    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4" podUID="973e3530-023a-4161-bda6-b54d52563c06" containerName="payload-processing" containerID="cri-o://fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104" gracePeriod=30
Sep 09 14:28:56.733892 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.733866    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:56.764455 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.764432    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:56.840217 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.840181    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-maas-api-tls\") pod \"c2daa202-2a81-45a7-9827-80215b8ed258\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") "
Sep 09 14:28:56.840390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.840254    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-metrics-certs\") pod \"c2daa202-2a81-45a7-9827-80215b8ed258\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") "
Sep 09 14:28:56.840390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.840306    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gcf7\" (UniqueName: \"kubernetes.io/projected/c2daa202-2a81-45a7-9827-80215b8ed258-kube-api-access-8gcf7\") pod \"c2daa202-2a81-45a7-9827-80215b8ed258\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") "
Sep 09 14:28:56.840390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.840340    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2daa202-2a81-45a7-9827-80215b8ed258-openshift-service-ca\") pod \"c2daa202-2a81-45a7-9827-80215b8ed258\" (UID: \"c2daa202-2a81-45a7-9827-80215b8ed258\") "
Sep 09 14:28:56.840576 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.840418    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973e3530-023a-4161-bda6-b54d52563c06-plugins-config-volume\") pod \"973e3530-023a-4161-bda6-b54d52563c06\" (UID: \"973e3530-023a-4161-bda6-b54d52563c06\") "
Sep 09 14:28:56.840576 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.840459    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz7sv\" (UniqueName: \"kubernetes.io/projected/973e3530-023a-4161-bda6-b54d52563c06-kube-api-access-xz7sv\") pod \"973e3530-023a-4161-bda6-b54d52563c06\" (UID: \"973e3530-023a-4161-bda6-b54d52563c06\") "
Sep 09 14:28:56.841000 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.840937    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/973e3530-023a-4161-bda6-b54d52563c06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "973e3530-023a-4161-bda6-b54d52563c06" (UID: "973e3530-023a-4161-bda6-b54d52563c06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:28:56.841000 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.840945    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2daa202-2a81-45a7-9827-80215b8ed258-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2daa202-2a81-45a7-9827-80215b8ed258" (UID: "c2daa202-2a81-45a7-9827-80215b8ed258"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:28:56.842493 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.842469    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2daa202-2a81-45a7-9827-80215b8ed258-kube-api-access-8gcf7" (OuterVolumeSpecName: "kube-api-access-8gcf7") pod "c2daa202-2a81-45a7-9827-80215b8ed258" (UID: "c2daa202-2a81-45a7-9827-80215b8ed258"). InnerVolumeSpecName "kube-api-access-8gcf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:28:56.842714 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.842688    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973e3530-023a-4161-bda6-b54d52563c06-kube-api-access-xz7sv" (OuterVolumeSpecName: "kube-api-access-xz7sv") pod "973e3530-023a-4161-bda6-b54d52563c06" (UID: "973e3530-023a-4161-bda6-b54d52563c06"). InnerVolumeSpecName "kube-api-access-xz7sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:28:56.842850 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.842828    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c2daa202-2a81-45a7-9827-80215b8ed258" (UID: "c2daa202-2a81-45a7-9827-80215b8ed258"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:28:56.842912 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.842845    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2daa202-2a81-45a7-9827-80215b8ed258" (UID: "c2daa202-2a81-45a7-9827-80215b8ed258"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:28:56.941499 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.941420    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973e3530-023a-4161-bda6-b54d52563c06-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:56.941499 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.941452    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz7sv\" (UniqueName: \"kubernetes.io/projected/973e3530-023a-4161-bda6-b54d52563c06-kube-api-access-xz7sv\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:56.941499 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.941468    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:56.941499 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.941482    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2daa202-2a81-45a7-9827-80215b8ed258-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:56.941499 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.941495    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gcf7\" (UniqueName: \"kubernetes.io/projected/c2daa202-2a81-45a7-9827-80215b8ed258-kube-api-access-8gcf7\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:56.941807 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:56.941508    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2daa202-2a81-45a7-9827-80215b8ed258-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:57.072981 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.072945    2532 generic.go:356] "Generic (PLEG): container finished" podID="973e3530-023a-4161-bda6-b54d52563c06" containerID="fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104" exitCode=0
Sep 09 14:28:57.073198 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.072997    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"
Sep 09 14:28:57.073198 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.073014    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4" event={"ID":"973e3530-023a-4161-bda6-b54d52563c06","Type":"ContainerDied","Data":"fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104"}
Sep 09 14:28:57.073198 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.073067    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4" event={"ID":"973e3530-023a-4161-bda6-b54d52563c06","Type":"ContainerDied","Data":"3ea7afb3335fdc3ba9238d073c2396eac37b94b69a0b6bb8d755c3fb1e12f534"}
Sep 09 14:28:57.073198 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.073123    2532 scope.go:117] "RemoveContainer" containerID="fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104"
Sep 09 14:28:57.074529 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.074504    2532 generic.go:356] "Generic (PLEG): container finished" podID="c2daa202-2a81-45a7-9827-80215b8ed258" containerID="fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19" exitCode=0
Sep 09 14:28:57.074641 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.074540    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"
Sep 09 14:28:57.074641 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.074580    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b" event={"ID":"c2daa202-2a81-45a7-9827-80215b8ed258","Type":"ContainerDied","Data":"fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19"}
Sep 09 14:28:57.074641 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.074624    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b" event={"ID":"c2daa202-2a81-45a7-9827-80215b8ed258","Type":"ContainerDied","Data":"d245a9a114dd29a86ca8e7c3978bc1e62e87fcfa8106c574ffb64fc882ece39e"}
Sep 09 14:28:57.082823 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.082805    2532 scope.go:117] "RemoveContainer" containerID="fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104"
Sep 09 14:28:57.083112 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:28:57.083081    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104\": container with ID starting with fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104 not found: ID does not exist" containerID="fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104"
Sep 09 14:28:57.083198 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.083120    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104"} err="failed to get container status \"fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104\": rpc error: code = NotFound desc = could not find container \"fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104\": container with ID starting with fdb3a6572ce86b8e13379c396146a4fde1072cc72c71cec1dc42be863cc27104 not found: ID does not exist"
Sep 09 14:28:57.083198 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.083136    2532 scope.go:117] "RemoveContainer" containerID="fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19"
Sep 09 14:28:57.091655 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.091640    2532 scope.go:117] "RemoveContainer" containerID="fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19"
Sep 09 14:28:57.091912 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:28:57.091893    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19\": container with ID starting with fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19 not found: ID does not exist" containerID="fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19"
Sep 09 14:28:57.091979 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.091921    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19"} err="failed to get container status \"fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19\": rpc error: code = NotFound desc = could not find container \"fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19\": container with ID starting with fa8fbde01ebc2f0d91c676958c6a348bed853560be824e1b08218121ee8c9d19 not found: ID does not exist"
Sep 09 14:28:57.097693 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.097672    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"]
Sep 09 14:28:57.101818 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.101800    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94c06b6-5b8c4f954f-x2k2b"]
Sep 09 14:28:57.112818 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.112799    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"]
Sep 09 14:28:57.116372 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:57.116353    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d94c06b6-86664bc8d9-mxsb4"]
Sep 09 14:28:58.443904 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.443870    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"]
Sep 09 14:28:58.444365 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.444120    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g" podUID="12634c2a-21aa-4f35-84e3-f58ee9bc07f8" containerName="payload-pre-processing" containerID="cri-o://14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27" gracePeriod=30
Sep 09 14:28:58.697499 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.697444    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:28:58.735219 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.735192    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="973e3530-023a-4161-bda6-b54d52563c06" path="/var/lib/kubelet/pods/973e3530-023a-4161-bda6-b54d52563c06/volumes"
Sep 09 14:28:58.735597 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.735585    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2daa202-2a81-45a7-9827-80215b8ed258" path="/var/lib/kubelet/pods/c2daa202-2a81-45a7-9827-80215b8ed258/volumes"
Sep 09 14:28:58.860002 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.859972    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-plugins-config-volume\") pod \"12634c2a-21aa-4f35-84e3-f58ee9bc07f8\" (UID: \"12634c2a-21aa-4f35-84e3-f58ee9bc07f8\") "
Sep 09 14:28:58.860175 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.860090    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xtbv\" (UniqueName: \"kubernetes.io/projected/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-kube-api-access-4xtbv\") pod \"12634c2a-21aa-4f35-84e3-f58ee9bc07f8\" (UID: \"12634c2a-21aa-4f35-84e3-f58ee9bc07f8\") "
Sep 09 14:28:58.860393 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.860369    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12634c2a-21aa-4f35-84e3-f58ee9bc07f8" (UID: "12634c2a-21aa-4f35-84e3-f58ee9bc07f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:28:58.862076 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.862042    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-kube-api-access-4xtbv" (OuterVolumeSpecName: "kube-api-access-4xtbv") pod "12634c2a-21aa-4f35-84e3-f58ee9bc07f8" (UID: "12634c2a-21aa-4f35-84e3-f58ee9bc07f8"). InnerVolumeSpecName "kube-api-access-4xtbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:28:58.961593 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.961522    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xtbv\" (UniqueName: \"kubernetes.io/projected/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-kube-api-access-4xtbv\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:58.961593 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:58.961551    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12634c2a-21aa-4f35-84e3-f58ee9bc07f8-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:28:59.090548 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:59.090481    2532 generic.go:356] "Generic (PLEG): container finished" podID="12634c2a-21aa-4f35-84e3-f58ee9bc07f8" containerID="14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27" exitCode=0
Sep 09 14:28:59.090548 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:59.090519    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"
Sep 09 14:28:59.090758 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:59.090567    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g" event={"ID":"12634c2a-21aa-4f35-84e3-f58ee9bc07f8","Type":"ContainerDied","Data":"14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27"}
Sep 09 14:28:59.090758 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:59.090617    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g" event={"ID":"12634c2a-21aa-4f35-84e3-f58ee9bc07f8","Type":"ContainerDied","Data":"15476b258aebfad403ec840966c09e1f34cd355be6611f05dfeafe9d6b46f5fa"}
Sep 09 14:28:59.090758 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:59.090639    2532 scope.go:117] "RemoveContainer" containerID="14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27"
Sep 09 14:28:59.099780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:59.099764    2532 scope.go:117] "RemoveContainer" containerID="14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27"
Sep 09 14:28:59.100004 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:28:59.099986    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27\": container with ID starting with 14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27 not found: ID does not exist" containerID="14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27"
Sep 09 14:28:59.100055 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:59.100010    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27"} err="failed to get container status \"14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27\": rpc error: code = NotFound desc = could not find container \"14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27\": container with ID starting with 14b5d4f59fb0977255ebb3553cef8da39b0c59fa4e57fd91bdc10dfb3a0d1a27 not found: ID does not exist"
Sep 09 14:28:59.131292 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:59.131268    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"]
Sep 09 14:28:59.151516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:28:59.151492    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d85f87-d6c9d9fdb-w4m2g"]
Sep 09 14:29:00.734581 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:00.734545    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12634c2a-21aa-4f35-84e3-f58ee9bc07f8" path="/var/lib/kubelet/pods/12634c2a-21aa-4f35-84e3-f58ee9bc07f8/volumes"
Sep 09 14:29:11.768076 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768036    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"]
Sep 09 14:29:11.768643 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768629    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="973e3530-023a-4161-bda6-b54d52563c06" containerName="payload-processing"
Sep 09 14:29:11.768716 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768653    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="973e3530-023a-4161-bda6-b54d52563c06" containerName="payload-processing"
Sep 09 14:29:11.768716 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768672    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a390b7-6abc-440b-9513-5e06d611abd7" containerName="payload-pre-processing"
Sep 09 14:29:11.768716 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768712    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a390b7-6abc-440b-9513-5e06d611abd7" containerName="payload-pre-processing"
Sep 09 14:29:11.768872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768729    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12634c2a-21aa-4f35-84e3-f58ee9bc07f8" containerName="payload-pre-processing"
Sep 09 14:29:11.768872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768739    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="12634c2a-21aa-4f35-84e3-f58ee9bc07f8" containerName="payload-pre-processing"
Sep 09 14:29:11.768872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768756    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ec8ea3-fb1b-4503-9093-279a6dbf462c" containerName="main"
Sep 09 14:29:11.768872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768764    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ec8ea3-fb1b-4503-9093-279a6dbf462c" containerName="main"
Sep 09 14:29:11.768872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768784    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a3aa9c8-ea76-43dc-b31a-78714701aba9" containerName="main"
Sep 09 14:29:11.768872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768793    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a3aa9c8-ea76-43dc-b31a-78714701aba9" containerName="main"
Sep 09 14:29:11.768872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768808    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2daa202-2a81-45a7-9827-80215b8ed258" containerName="maas-api"
Sep 09 14:29:11.768872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768816    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2daa202-2a81-45a7-9827-80215b8ed258" containerName="maas-api"
Sep 09 14:29:11.768872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768829    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9" containerName="maas-api"
Sep 09 14:29:11.768872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768837    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9" containerName="maas-api"
Sep 09 14:29:11.769374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768942    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2daa202-2a81-45a7-9827-80215b8ed258" containerName="maas-api"
Sep 09 14:29:11.769374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768956    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a390b7-6abc-440b-9513-5e06d611abd7" containerName="payload-pre-processing"
Sep 09 14:29:11.769374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768967    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a3aa9c8-ea76-43dc-b31a-78714701aba9" containerName="main"
Sep 09 14:29:11.769374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768978    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6c4f81-a100-4d0f-bdcb-6144b27d3ab9" containerName="maas-api"
Sep 09 14:29:11.769374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768987    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="973e3530-023a-4161-bda6-b54d52563c06" containerName="payload-processing"
Sep 09 14:29:11.769374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.768997    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ec8ea3-fb1b-4503-9093-279a6dbf462c" containerName="main"
Sep 09 14:29:11.769374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.769011    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="12634c2a-21aa-4f35-84e3-f58ee9bc07f8" containerName="payload-pre-processing"
Sep 09 14:29:11.773969 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.773947    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:29:11.776432 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.776411    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e565aa-dockercfg-wkk7q\""
Sep 09 14:29:11.776540 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.776481    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e565aa\""
Sep 09 14:29:11.778033 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.777940    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq8n6\" (UniqueName: \"kubernetes.io/projected/c4e72623-88f1-4eff-8058-8c60f72e295e-kube-api-access-gq8n6\") pod \"payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26\" (UID: \"c4e72623-88f1-4eff-8058-8c60f72e295e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:29:11.778362 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.778339    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e72623-88f1-4eff-8058-8c60f72e295e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26\" (UID: \"c4e72623-88f1-4eff-8058-8c60f72e295e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:29:11.781217 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.781194    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"]
Sep 09 14:29:11.879417 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.879392    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e72623-88f1-4eff-8058-8c60f72e295e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26\" (UID: \"c4e72623-88f1-4eff-8058-8c60f72e295e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:29:11.879551 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.879520    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq8n6\" (UniqueName: \"kubernetes.io/projected/c4e72623-88f1-4eff-8058-8c60f72e295e-kube-api-access-gq8n6\") pod \"payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26\" (UID: \"c4e72623-88f1-4eff-8058-8c60f72e295e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:29:11.880215 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.880188    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e72623-88f1-4eff-8058-8c60f72e295e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26\" (UID: \"c4e72623-88f1-4eff-8058-8c60f72e295e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:29:11.887741 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:11.887713    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq8n6\" (UniqueName: \"kubernetes.io/projected/c4e72623-88f1-4eff-8058-8c60f72e295e-kube-api-access-gq8n6\") pod \"payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26\" (UID: \"c4e72623-88f1-4eff-8058-8c60f72e295e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:29:12.086340 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:12.086278    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:29:12.358528 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:12.358500    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"]
Sep 09 14:29:12.359968 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:29:12.359941    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e72623_88f1_4eff_8058_8c60f72e295e.slice/crio-851e808a313ae6a49cbcd8050b38dddaa53dbb6b036b6eba30dc1810df84abec WatchSource:0}: Error finding container 851e808a313ae6a49cbcd8050b38dddaa53dbb6b036b6eba30dc1810df84abec: Status 404 returned error can't find the container with id 851e808a313ae6a49cbcd8050b38dddaa53dbb6b036b6eba30dc1810df84abec
Sep 09 14:29:13.163731 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:13.163698    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26" event={"ID":"c4e72623-88f1-4eff-8058-8c60f72e295e","Type":"ContainerStarted","Data":"36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050"}
Sep 09 14:29:13.164094 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:13.163748    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26" event={"ID":"c4e72623-88f1-4eff-8058-8c60f72e295e","Type":"ContainerStarted","Data":"851e808a313ae6a49cbcd8050b38dddaa53dbb6b036b6eba30dc1810df84abec"}
Sep 09 14:29:13.164094 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:13.163777    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:29:13.183838 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:13.183794    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26" podStartSLOduration=2.18377853 podStartE2EDuration="2.18377853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:29:13.181739162 +0000 UTC m=+1086.944534402" watchObservedRunningTime="2026-09-09 14:29:13.18377853 +0000 UTC m=+1086.946573770"
Sep 09 14:29:24.170682 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:24.170654    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:29:24.904666 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:24.904636    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"]
Sep 09 14:29:24.908136 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:24.908116    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:24.910855 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:24.910825    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-cbb7fb-serving-cert\""
Sep 09 14:29:24.918955 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:24.918932    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"]
Sep 09 14:29:24.988988 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:24.988957    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8v76\" (UniqueName: \"kubernetes.io/projected/5f273552-4024-4df1-a8d9-a068edf2c89b-kube-api-access-w8v76\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:24.989128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:24.988996    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f273552-4024-4df1-a8d9-a068edf2c89b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:24.989128 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:24.989032    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:24.989221 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:24.989160    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:25.090329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:25.090298    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8v76\" (UniqueName: \"kubernetes.io/projected/5f273552-4024-4df1-a8d9-a068edf2c89b-kube-api-access-w8v76\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:25.090436 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:25.090333    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f273552-4024-4df1-a8d9-a068edf2c89b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:25.090436 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:25.090366    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:25.090436 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:25.090408    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:25.090541 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:29:25.090520    2532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-serving-cert: secret "maas-api-e2e-shared-a-w2-cbb7fb-serving-cert" not found
Sep 09 14:29:25.090600 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:29:25.090588    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-maas-api-tls podName:5f273552-4024-4df1-a8d9-a068edf2c89b nodeName:}" failed. No retries permitted until 2026-09-09 14:29:25.590568819 +0000 UTC m=+1099.353364051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj" (UID: "5f273552-4024-4df1-a8d9-a068edf2c89b") : secret "maas-api-e2e-shared-a-w2-cbb7fb-serving-cert" not found
Sep 09 14:29:25.091064 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:25.091038    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f273552-4024-4df1-a8d9-a068edf2c89b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:25.092632 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:25.092605    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:25.101269 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:25.101250    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8v76\" (UniqueName: \"kubernetes.io/projected/5f273552-4024-4df1-a8d9-a068edf2c89b-kube-api-access-w8v76\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:25.594885 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:25.594852    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:25.597542 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:25.597516    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:25.820017 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:25.819991    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:26.090481 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:26.090456    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"]
Sep 09 14:29:26.092659 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:29:26.092626    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f273552_4024_4df1_a8d9_a068edf2c89b.slice/crio-e3fde47768d644dea7e7161e893167a0dfc79981bd44a384bfa9aa28893d60ca WatchSource:0}: Error finding container e3fde47768d644dea7e7161e893167a0dfc79981bd44a384bfa9aa28893d60ca: Status 404 returned error can't find the container with id e3fde47768d644dea7e7161e893167a0dfc79981bd44a384bfa9aa28893d60ca
Sep 09 14:29:26.230464 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:26.230438    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj" event={"ID":"5f273552-4024-4df1-a8d9-a068edf2c89b","Type":"ContainerStarted","Data":"e3fde47768d644dea7e7161e893167a0dfc79981bd44a384bfa9aa28893d60ca"}
Sep 09 14:29:28.241436 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:28.241399    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj" event={"ID":"5f273552-4024-4df1-a8d9-a068edf2c89b","Type":"ContainerStarted","Data":"add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440"}
Sep 09 14:29:28.241787 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:28.241544    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:28.259481 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:28.259438    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj" podStartSLOduration=3.012972413 podStartE2EDuration="4.25942396s" totalImagesPullingTime="1.246451547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:29:28.258404083 +0000 UTC m=+1102.021199323" watchObservedRunningTime="2026-09-09 14:29:28.25942396 +0000 UTC m=+1102.022219201"
Sep 09 14:29:34.252016 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:34.251989    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:29:39.982328 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:39.982294    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"]
Sep 09 14:29:39.986997 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:39.986977    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:39.989263 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:39.989241    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c5e2d3-serving-cert\""
Sep 09 14:29:39.997440 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:39.997411    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"]
Sep 09 14:29:40.116023 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.115994    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmcv7\" (UniqueName: \"kubernetes.io/projected/c539c285-1bb3-4ed3-9703-a1e2862d499d-kube-api-access-bmcv7\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.116179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.116039    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.116179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.116061    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c539c285-1bb3-4ed3-9703-a1e2862d499d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.116179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.116126    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.217457 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.217423    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmcv7\" (UniqueName: \"kubernetes.io/projected/c539c285-1bb3-4ed3-9703-a1e2862d499d-kube-api-access-bmcv7\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.217587 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.217472    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.217587 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.217506    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c539c285-1bb3-4ed3-9703-a1e2862d499d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.217587 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.217530    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.218197 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.218177    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c539c285-1bb3-4ed3-9703-a1e2862d499d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.219819 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.219798    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.219894 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.219841    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.225202 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.225183    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmcv7\" (UniqueName: \"kubernetes.io/projected/c539c285-1bb3-4ed3-9703-a1e2862d499d-kube-api-access-bmcv7\") pod \"maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.299489 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.299442    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:40.574008 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.573982    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"]
Sep 09 14:29:40.575827 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:29:40.575801    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc539c285_1bb3_4ed3_9703_a1e2862d499d.slice/crio-9caa15ef44f2e3d4286cf4aa1a9c50940ce352e1086e45a1cb0d94b0ebbccab3 WatchSource:0}: Error finding container 9caa15ef44f2e3d4286cf4aa1a9c50940ce352e1086e45a1cb0d94b0ebbccab3: Status 404 returned error can't find the container with id 9caa15ef44f2e3d4286cf4aa1a9c50940ce352e1086e45a1cb0d94b0ebbccab3
Sep 09 14:29:40.631497 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.631469    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"]
Sep 09 14:29:40.635816 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.635797    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:29:40.638514 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.638495    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c5e2d3-dockercfg-dkxxk\""
Sep 09 14:29:40.638599 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.638545    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c5e2d3\""
Sep 09 14:29:40.647661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.647632    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"]
Sep 09 14:29:40.722997 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.722969    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vrm5\" (UniqueName: \"kubernetes.io/projected/7a24e00d-c292-434a-86bc-ad1c3a758a17-kube-api-access-4vrm5\") pod \"payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8\" (UID: \"7a24e00d-c292-434a-86bc-ad1c3a758a17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:29:40.723130 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.723039    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a24e00d-c292-434a-86bc-ad1c3a758a17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8\" (UID: \"7a24e00d-c292-434a-86bc-ad1c3a758a17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:29:40.823618 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.823549    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vrm5\" (UniqueName: \"kubernetes.io/projected/7a24e00d-c292-434a-86bc-ad1c3a758a17-kube-api-access-4vrm5\") pod \"payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8\" (UID: \"7a24e00d-c292-434a-86bc-ad1c3a758a17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:29:40.823718 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.823634    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a24e00d-c292-434a-86bc-ad1c3a758a17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8\" (UID: \"7a24e00d-c292-434a-86bc-ad1c3a758a17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:29:40.824285 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.824268    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a24e00d-c292-434a-86bc-ad1c3a758a17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8\" (UID: \"7a24e00d-c292-434a-86bc-ad1c3a758a17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:29:40.833220 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.833199    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vrm5\" (UniqueName: \"kubernetes.io/projected/7a24e00d-c292-434a-86bc-ad1c3a758a17-kube-api-access-4vrm5\") pod \"payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8\" (UID: \"7a24e00d-c292-434a-86bc-ad1c3a758a17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:29:40.949413 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:40.949383    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:29:41.229987 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:41.229954    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"]
Sep 09 14:29:41.230363 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:29:41.230189    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a24e00d_c292_434a_86bc_ad1c3a758a17.slice/crio-9d33355fb5db8271e7ae94ef7b1d18b974990d8576d19340511e97b732b1d8a8 WatchSource:0}: Error finding container 9d33355fb5db8271e7ae94ef7b1d18b974990d8576d19340511e97b732b1d8a8: Status 404 returned error can't find the container with id 9d33355fb5db8271e7ae94ef7b1d18b974990d8576d19340511e97b732b1d8a8
Sep 09 14:29:41.306723 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:41.306679    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8" event={"ID":"7a24e00d-c292-434a-86bc-ad1c3a758a17","Type":"ContainerStarted","Data":"19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff"}
Sep 09 14:29:41.306860 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:41.306738    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8" event={"ID":"7a24e00d-c292-434a-86bc-ad1c3a758a17","Type":"ContainerStarted","Data":"9d33355fb5db8271e7ae94ef7b1d18b974990d8576d19340511e97b732b1d8a8"}
Sep 09 14:29:41.306860 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:41.306769    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:29:41.308525 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:41.308502    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl" event={"ID":"c539c285-1bb3-4ed3-9703-a1e2862d499d","Type":"ContainerStarted","Data":"f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2"}
Sep 09 14:29:41.308643 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:41.308550    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl" event={"ID":"c539c285-1bb3-4ed3-9703-a1e2862d499d","Type":"ContainerStarted","Data":"9caa15ef44f2e3d4286cf4aa1a9c50940ce352e1086e45a1cb0d94b0ebbccab3"}
Sep 09 14:29:41.308643 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:41.308591    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:41.325610 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:41.325568    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8" podStartSLOduration=1.325554755 podStartE2EDuration="1.325554755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:29:41.324806915 +0000 UTC m=+1115.087602155" watchObservedRunningTime="2026-09-09 14:29:41.325554755 +0000 UTC m=+1115.088349994"
Sep 09 14:29:41.345853 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:41.345815    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl" podStartSLOduration=1.941274791 podStartE2EDuration="2.345802724s" totalImagesPullingTime="404.527933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:29:41.344841626 +0000 UTC m=+1115.107636867" watchObservedRunningTime="2026-09-09 14:29:41.345802724 +0000 UTC m=+1115.108598416"
Sep 09 14:29:47.323638 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:47.323605    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:29:52.319810 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:52.319779    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:29:53.997951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:53.997798    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"]
Sep 09 14:29:54.002736 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.002703    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.005040 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.005014    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-fecaf9-serving-cert\""
Sep 09 14:29:54.012837 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.012728    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"]
Sep 09 14:29:54.140883 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.140850    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk8lx\" (UniqueName: \"kubernetes.io/projected/9a9e8981-c92a-4858-98e5-396f3bb05538-kube-api-access-sk8lx\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.141044 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.140970    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9e8981-c92a-4858-98e5-396f3bb05538-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.141044 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.141011    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.141044 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.141029    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.241763 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.241735    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk8lx\" (UniqueName: \"kubernetes.io/projected/9a9e8981-c92a-4858-98e5-396f3bb05538-kube-api-access-sk8lx\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.241937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.241799    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9e8981-c92a-4858-98e5-396f3bb05538-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.241937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.241824    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.241937 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.241842    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.242153 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:29:54.241940    2532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-serving-cert: secret "maas-api-e2e-shared-b-w2-fecaf9-serving-cert" not found
Sep 09 14:29:54.242153 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:29:54.242016    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-maas-api-tls podName:9a9e8981-c92a-4858-98e5-396f3bb05538 nodeName:}" failed. No retries permitted until 2026-09-09 14:29:54.741988959 +0000 UTC m=+1128.504784179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-maas-api-tls") pod "maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8" (UID: "9a9e8981-c92a-4858-98e5-396f3bb05538") : secret "maas-api-e2e-shared-b-w2-fecaf9-serving-cert" not found
Sep 09 14:29:54.242530 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.242508    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9e8981-c92a-4858-98e5-396f3bb05538-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.244180 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.244161    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.252914 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.252866    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk8lx\" (UniqueName: \"kubernetes.io/projected/9a9e8981-c92a-4858-98e5-396f3bb05538-kube-api-access-sk8lx\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.745285 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.745255    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.747504 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.747481    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.819527 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.819498    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"]
Sep 09 14:29:54.823139 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.823118    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:29:54.825604 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.825578    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-fecaf9\""
Sep 09 14:29:54.825706 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.825584    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-fecaf9-dockercfg-spg78\""
Sep 09 14:29:54.833141 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.833119    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"]
Sep 09 14:29:54.917150 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.917126    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:54.947178 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.947146    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c9851c-c9c1-476d-86e1-643f04f2c4f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7\" (UID: \"84c9851c-c9c1-476d-86e1-643f04f2c4f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:29:54.947321 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:54.947301    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vhh\" (UniqueName: \"kubernetes.io/projected/84c9851c-c9c1-476d-86e1-643f04f2c4f2-kube-api-access-r2vhh\") pod \"payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7\" (UID: \"84c9851c-c9c1-476d-86e1-643f04f2c4f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:29:55.048289 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:55.048219    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vhh\" (UniqueName: \"kubernetes.io/projected/84c9851c-c9c1-476d-86e1-643f04f2c4f2-kube-api-access-r2vhh\") pod \"payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7\" (UID: \"84c9851c-c9c1-476d-86e1-643f04f2c4f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:29:55.048289 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:55.048276    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c9851c-c9c1-476d-86e1-643f04f2c4f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7\" (UID: \"84c9851c-c9c1-476d-86e1-643f04f2c4f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:29:55.048911 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:55.048893    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c9851c-c9c1-476d-86e1-643f04f2c4f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7\" (UID: \"84c9851c-c9c1-476d-86e1-643f04f2c4f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:29:55.059480 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:55.059454    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vhh\" (UniqueName: \"kubernetes.io/projected/84c9851c-c9c1-476d-86e1-643f04f2c4f2-kube-api-access-r2vhh\") pod \"payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7\" (UID: \"84c9851c-c9c1-476d-86e1-643f04f2c4f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:29:55.134849 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:55.134827    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:29:55.188475 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:55.188414    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"]
Sep 09 14:29:55.193130 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:29:55.193081    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9e8981_c92a_4858_98e5_396f3bb05538.slice/crio-a8a980928259898fb097a099cde3d0bdd1a9e83f064e639bda095eb279ce186c WatchSource:0}: Error finding container a8a980928259898fb097a099cde3d0bdd1a9e83f064e639bda095eb279ce186c: Status 404 returned error can't find the container with id a8a980928259898fb097a099cde3d0bdd1a9e83f064e639bda095eb279ce186c
Sep 09 14:29:55.393866 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:55.393800    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8" event={"ID":"9a9e8981-c92a-4858-98e5-396f3bb05538","Type":"ContainerStarted","Data":"a8a980928259898fb097a099cde3d0bdd1a9e83f064e639bda095eb279ce186c"}
Sep 09 14:29:55.405351 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:29:55.405326    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c9851c_c9c1_476d_86e1_643f04f2c4f2.slice/crio-b9a495a5f466631069f4e7495a17a49943148f867e037ca06f46b9e4acb6be8a WatchSource:0}: Error finding container b9a495a5f466631069f4e7495a17a49943148f867e037ca06f46b9e4acb6be8a: Status 404 returned error can't find the container with id b9a495a5f466631069f4e7495a17a49943148f867e037ca06f46b9e4acb6be8a
Sep 09 14:29:55.410351 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:55.410326    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"]
Sep 09 14:29:56.400566 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:56.400529    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8" event={"ID":"9a9e8981-c92a-4858-98e5-396f3bb05538","Type":"ContainerStarted","Data":"161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2"}
Sep 09 14:29:56.401023 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:56.400597    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:29:56.402169 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:56.402145    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7" event={"ID":"84c9851c-c9c1-476d-86e1-643f04f2c4f2","Type":"ContainerStarted","Data":"e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e"}
Sep 09 14:29:56.402275 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:56.402187    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7" event={"ID":"84c9851c-c9c1-476d-86e1-643f04f2c4f2","Type":"ContainerStarted","Data":"b9a495a5f466631069f4e7495a17a49943148f867e037ca06f46b9e4acb6be8a"}
Sep 09 14:29:56.402275 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:56.402211    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:29:56.418286 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:56.418243    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8" podStartSLOduration=3.042230422 podStartE2EDuration="3.418228797s" totalImagesPullingTime="375.998375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:29:56.417017681 +0000 UTC m=+1130.179812927" watchObservedRunningTime="2026-09-09 14:29:56.418228797 +0000 UTC m=+1130.181024036"
Sep 09 14:29:56.433488 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:29:56.433448    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7" podStartSLOduration=2.433436101 podStartE2EDuration="2.433436101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:29:56.432032492 +0000 UTC m=+1130.194827734" watchObservedRunningTime="2026-09-09 14:29:56.433436101 +0000 UTC m=+1130.196231340"
Sep 09 14:30:02.413740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:02.413708    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:30:07.409179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:07.409150    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:30:35.819671 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:35.819595    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"]
Sep 09 14:30:35.823319 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:35.823298    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:35.825953 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:35.825933    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5b36371b-serving-cert\""
Sep 09 14:30:35.834830 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:35.834807    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"]
Sep 09 14:30:35.915422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:35.915390    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-openshift-service-ca\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:35.915422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:35.915421    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-metrics-certs\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:35.915602 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:35.915442    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-maas-api-tls\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:35.915602 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:35.915534    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77n4z\" (UniqueName: \"kubernetes.io/projected/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-kube-api-access-77n4z\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:36.016056 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:36.016029    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77n4z\" (UniqueName: \"kubernetes.io/projected/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-kube-api-access-77n4z\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:36.016215 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:36.016125    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-openshift-service-ca\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:36.016281 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:36.016259    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-metrics-certs\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:36.016346 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:36.016306    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-maas-api-tls\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:36.016479 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:30:36.016461    2532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-serving-cert: secret "maas-api-e2e-mt-5b36371b-serving-cert" not found
Sep 09 14:30:36.016530 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:30:36.016520    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-maas-api-tls podName:3fbd17d5-dd82-436a-9e7c-41588d2eccf8 nodeName:}" failed. No retries permitted until 2026-09-09 14:30:36.516504928 +0000 UTC m=+1170.279300146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-maas-api-tls") pod "maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j" (UID: "3fbd17d5-dd82-436a-9e7c-41588d2eccf8") : secret "maas-api-e2e-mt-5b36371b-serving-cert" not found
Sep 09 14:30:36.016729 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:36.016711    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-openshift-service-ca\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:36.018511 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:36.018486    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-metrics-certs\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:36.024667 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:36.024646    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77n4z\" (UniqueName: \"kubernetes.io/projected/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-kube-api-access-77n4z\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:36.521056 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:36.521018    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-maas-api-tls\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:36.523257 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:36.523227    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-maas-api-tls\") pod \"maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:36.735664 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:36.735639    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:37.007502 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:37.007479    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"]
Sep 09 14:30:37.009188 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:30:37.009161    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbd17d5_dd82_436a_9e7c_41588d2eccf8.slice/crio-f8511c43f92d4643f4be9fe1c83d6ff630ac72e6239de9145f8bd91b6f2bc639 WatchSource:0}: Error finding container f8511c43f92d4643f4be9fe1c83d6ff630ac72e6239de9145f8bd91b6f2bc639: Status 404 returned error can't find the container with id f8511c43f92d4643f4be9fe1c83d6ff630ac72e6239de9145f8bd91b6f2bc639
Sep 09 14:30:37.632395 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:37.632357    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j" event={"ID":"3fbd17d5-dd82-436a-9e7c-41588d2eccf8","Type":"ContainerStarted","Data":"f8511c43f92d4643f4be9fe1c83d6ff630ac72e6239de9145f8bd91b6f2bc639"}
Sep 09 14:30:38.639913 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:38.639875    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j" event={"ID":"3fbd17d5-dd82-436a-9e7c-41588d2eccf8","Type":"ContainerStarted","Data":"ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082"}
Sep 09 14:30:38.640314 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:38.639972    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:30:38.657453 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:38.657380    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j" podStartSLOduration=2.70882853 podStartE2EDuration="3.65736536s" totalImagesPullingTime="948.53683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:30:38.656212967 +0000 UTC m=+1172.419008208" watchObservedRunningTime="2026-09-09 14:30:38.65736536 +0000 UTC m=+1172.420160599"
Sep 09 14:30:44.650533 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:30:44.650501    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:31:06.740261 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:06.740234    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:31:06.742515 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:06.742493    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:31:21.693631 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.693541    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"]
Sep 09 14:31:21.697495 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.697473    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.700713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.700691    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5b36371b\"/\"e2e-lifecycle-model-5b36371b-kserve-self-signed-certs\""
Sep 09 14:31:21.700831 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.700727    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5b36371b\"/\"kube-root-ca.crt\""
Sep 09 14:31:21.700898 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.700696    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5b36371b\"/\"openshift-service-ca.crt\""
Sep 09 14:31:21.700898 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.700696    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5b36371b\"/\"default-dockercfg-mpjnm\""
Sep 09 14:31:21.709337 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.709316    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"]
Sep 09 14:31:21.819893 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.819864    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-home\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.820041 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.819906    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-model-cache\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.820041 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.819925    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zch5f\" (UniqueName: \"kubernetes.io/projected/aac3ff2b-496a-45ab-a790-fd96875774e6-kube-api-access-zch5f\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.820041 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.819986    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac3ff2b-496a-45ab-a790-fd96875774e6-tls-certs\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.820041 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.820027    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-dshm\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.820216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.820077    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-tmp-dir\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.921639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.921545    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-dshm\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.921639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.921598    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-tmp-dir\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.926956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.921653    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-home\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.926956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.921681    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-model-cache\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.926956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.921700    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zch5f\" (UniqueName: \"kubernetes.io/projected/aac3ff2b-496a-45ab-a790-fd96875774e6-kube-api-access-zch5f\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.926956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.921828    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac3ff2b-496a-45ab-a790-fd96875774e6-tls-certs\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.926956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.922006    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-model-cache\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.926956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.922086    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-tmp-dir\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.926956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.922127    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-home\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.926956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.923688    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-dshm\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.926956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.924213    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac3ff2b-496a-45ab-a790-fd96875774e6-tls-certs\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:21.929568 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:21.929551    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zch5f\" (UniqueName: \"kubernetes.io/projected/aac3ff2b-496a-45ab-a790-fd96875774e6-kube-api-access-zch5f\") pod \"e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") " pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:22.007453 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:22.007422    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:22.279550 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:22.279477    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"]
Sep 09 14:31:22.282969 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:31:22.282942    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaac3ff2b_496a_45ab_a790_fd96875774e6.slice/crio-6f2b61f8a337844d59daf44b7c9b310e7b16e940d9fdcf2de9747c7cd17640eb WatchSource:0}: Error finding container 6f2b61f8a337844d59daf44b7c9b310e7b16e940d9fdcf2de9747c7cd17640eb: Status 404 returned error can't find the container with id 6f2b61f8a337844d59daf44b7c9b310e7b16e940d9fdcf2de9747c7cd17640eb
Sep 09 14:31:22.865123 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:22.865067    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l" event={"ID":"aac3ff2b-496a-45ab-a790-fd96875774e6","Type":"ContainerStarted","Data":"0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281"}
Sep 09 14:31:22.865524 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:22.865125    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l" event={"ID":"aac3ff2b-496a-45ab-a790-fd96875774e6","Type":"ContainerStarted","Data":"6f2b61f8a337844d59daf44b7c9b310e7b16e940d9fdcf2de9747c7cd17640eb"}
Sep 09 14:31:22.884112 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:22.884054    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l" podStartSLOduration=1.884039459 podStartE2EDuration="1.884039459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:31:22.882417417 +0000 UTC m=+1216.645212670" watchObservedRunningTime="2026-09-09 14:31:22.884039459 +0000 UTC m=+1216.646834746"
Sep 09 14:31:32.008245 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:32.008205    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:32.008245 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:32.008260    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:32.020840 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:32.020819    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:31:32.927142 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:31:32.927115    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:32:11.422856 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.422818    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"]
Sep 09 14:32:11.423341 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.423045    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j" podUID="3fbd17d5-dd82-436a-9e7c-41588d2eccf8" containerName="maas-api" containerID="cri-o://ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082" gracePeriod=30
Sep 09 14:32:11.678412 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.678351    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:32:11.762810 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.762783    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-metrics-certs\") pod \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") "
Sep 09 14:32:11.762970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.762845    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-maas-api-tls\") pod \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") "
Sep 09 14:32:11.762970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.762865    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77n4z\" (UniqueName: \"kubernetes.io/projected/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-kube-api-access-77n4z\") pod \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") "
Sep 09 14:32:11.762970 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.762897    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-openshift-service-ca\") pod \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\" (UID: \"3fbd17d5-dd82-436a-9e7c-41588d2eccf8\") "
Sep 09 14:32:11.763440 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.763267    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3fbd17d5-dd82-436a-9e7c-41588d2eccf8" (UID: "3fbd17d5-dd82-436a-9e7c-41588d2eccf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:32:11.763564 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.763540    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:32:11.765185 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.765155    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-kube-api-access-77n4z" (OuterVolumeSpecName: "kube-api-access-77n4z") pod "3fbd17d5-dd82-436a-9e7c-41588d2eccf8" (UID: "3fbd17d5-dd82-436a-9e7c-41588d2eccf8"). InnerVolumeSpecName "kube-api-access-77n4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:32:11.765329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.765307    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3fbd17d5-dd82-436a-9e7c-41588d2eccf8" (UID: "3fbd17d5-dd82-436a-9e7c-41588d2eccf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:32:11.765393 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.765338    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3fbd17d5-dd82-436a-9e7c-41588d2eccf8" (UID: "3fbd17d5-dd82-436a-9e7c-41588d2eccf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:32:11.864603 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.864577    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:32:11.864603 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.864599    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:32:11.864748 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:11.864608    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77n4z\" (UniqueName: \"kubernetes.io/projected/3fbd17d5-dd82-436a-9e7c-41588d2eccf8-kube-api-access-77n4z\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:32:12.122728 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:12.122697    2532 generic.go:356] "Generic (PLEG): container finished" podID="3fbd17d5-dd82-436a-9e7c-41588d2eccf8" containerID="ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082" exitCode=0
Sep 09 14:32:12.122896 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:12.122738    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"
Sep 09 14:32:12.122896 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:12.122782    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j" event={"ID":"3fbd17d5-dd82-436a-9e7c-41588d2eccf8","Type":"ContainerDied","Data":"ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082"}
Sep 09 14:32:12.122896 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:12.122839    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j" event={"ID":"3fbd17d5-dd82-436a-9e7c-41588d2eccf8","Type":"ContainerDied","Data":"f8511c43f92d4643f4be9fe1c83d6ff630ac72e6239de9145f8bd91b6f2bc639"}
Sep 09 14:32:12.122896 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:12.122868    2532 scope.go:117] "RemoveContainer" containerID="ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082"
Sep 09 14:32:12.133567 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:12.133553    2532 scope.go:117] "RemoveContainer" containerID="ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082"
Sep 09 14:32:12.133807 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:32:12.133791    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082\": container with ID starting with ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082 not found: ID does not exist" containerID="ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082"
Sep 09 14:32:12.133864 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:12.133814    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082"} err="failed to get container status \"ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082\": rpc error: code = NotFound desc = could not find container \"ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082\": container with ID starting with ae0995da21afc60b8852e49ed2eddfaf66b35dfdb5970416badae5494400c082 not found: ID does not exist"
Sep 09 14:32:12.145177 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:12.145153    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"]
Sep 09 14:32:12.149082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:12.149063    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b36371b-8465c5c85b-mhl7j"]
Sep 09 14:32:12.735882 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:12.735853    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fbd17d5-dd82-436a-9e7c-41588d2eccf8" path="/var/lib/kubelet/pods/3fbd17d5-dd82-436a-9e7c-41588d2eccf8/volumes"
Sep 09 14:32:21.484423 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:21.484389    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"]
Sep 09 14:32:36.504558 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.504511    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l" podUID="aac3ff2b-496a-45ab-a790-fd96875774e6" containerName="main" containerID="cri-o://0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281" gracePeriod=45
Sep 09 14:32:36.747329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.747305    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:32:36.884196 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.884112    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac3ff2b-496a-45ab-a790-fd96875774e6-tls-certs\") pod \"aac3ff2b-496a-45ab-a790-fd96875774e6\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") "
Sep 09 14:32:36.884333 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.884212    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zch5f\" (UniqueName: \"kubernetes.io/projected/aac3ff2b-496a-45ab-a790-fd96875774e6-kube-api-access-zch5f\") pod \"aac3ff2b-496a-45ab-a790-fd96875774e6\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") "
Sep 09 14:32:36.884333 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.884233    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-dshm\") pod \"aac3ff2b-496a-45ab-a790-fd96875774e6\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") "
Sep 09 14:32:36.884333 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.884267    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-home\") pod \"aac3ff2b-496a-45ab-a790-fd96875774e6\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") "
Sep 09 14:32:36.884333 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.884325    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-tmp-dir\") pod \"aac3ff2b-496a-45ab-a790-fd96875774e6\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") "
Sep 09 14:32:36.884562 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.884382    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-model-cache\") pod \"aac3ff2b-496a-45ab-a790-fd96875774e6\" (UID: \"aac3ff2b-496a-45ab-a790-fd96875774e6\") "
Sep 09 14:32:36.884562 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.884535    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-home" (OuterVolumeSpecName: "home") pod "aac3ff2b-496a-45ab-a790-fd96875774e6" (UID: "aac3ff2b-496a-45ab-a790-fd96875774e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:32:36.884720 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.884686    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "aac3ff2b-496a-45ab-a790-fd96875774e6" (UID: "aac3ff2b-496a-45ab-a790-fd96875774e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:32:36.884787 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.884722    2532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-home\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:32:36.884923 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.884893    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aac3ff2b-496a-45ab-a790-fd96875774e6" (UID: "aac3ff2b-496a-45ab-a790-fd96875774e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:32:36.886311 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.886288    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-dshm" (OuterVolumeSpecName: "dshm") pod "aac3ff2b-496a-45ab-a790-fd96875774e6" (UID: "aac3ff2b-496a-45ab-a790-fd96875774e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:32:36.886579 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.886550    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac3ff2b-496a-45ab-a790-fd96875774e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aac3ff2b-496a-45ab-a790-fd96875774e6" (UID: "aac3ff2b-496a-45ab-a790-fd96875774e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:32:36.886579 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.886563    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac3ff2b-496a-45ab-a790-fd96875774e6-kube-api-access-zch5f" (OuterVolumeSpecName: "kube-api-access-zch5f") pod "aac3ff2b-496a-45ab-a790-fd96875774e6" (UID: "aac3ff2b-496a-45ab-a790-fd96875774e6"). InnerVolumeSpecName "kube-api-access-zch5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:32:36.985771 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.985745    2532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-tmp-dir\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:32:36.985771 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.985769    2532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-model-cache\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:32:36.985918 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.985777    2532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac3ff2b-496a-45ab-a790-fd96875774e6-tls-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:32:36.985918 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.985786    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zch5f\" (UniqueName: \"kubernetes.io/projected/aac3ff2b-496a-45ab-a790-fd96875774e6-kube-api-access-zch5f\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:32:36.985918 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:36.985795    2532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aac3ff2b-496a-45ab-a790-fd96875774e6-dshm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:32:37.253512 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:37.253476    2532 generic.go:356] "Generic (PLEG): container finished" podID="aac3ff2b-496a-45ab-a790-fd96875774e6" containerID="0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281" exitCode=0
Sep 09 14:32:37.253704 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:37.253520    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"
Sep 09 14:32:37.253704 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:37.253558    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l" event={"ID":"aac3ff2b-496a-45ab-a790-fd96875774e6","Type":"ContainerDied","Data":"0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281"}
Sep 09 14:32:37.253704 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:37.253604    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l" event={"ID":"aac3ff2b-496a-45ab-a790-fd96875774e6","Type":"ContainerDied","Data":"6f2b61f8a337844d59daf44b7c9b310e7b16e940d9fdcf2de9747c7cd17640eb"}
Sep 09 14:32:37.253704 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:37.253627    2532 scope.go:117] "RemoveContainer" containerID="0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281"
Sep 09 14:32:37.262877 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:37.262863    2532 scope.go:117] "RemoveContainer" containerID="0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281"
Sep 09 14:32:37.263147 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:32:37.263129    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281\": container with ID starting with 0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281 not found: ID does not exist" containerID="0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281"
Sep 09 14:32:37.263196 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:37.263155    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281"} err="failed to get container status \"0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281\": rpc error: code = NotFound desc = could not find container \"0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281\": container with ID starting with 0a8ed03259b049a8b81e127dce6b480655857cb3fd762e41a8ddcc2f0ba42281 not found: ID does not exist"
Sep 09 14:32:37.274811 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:37.274791    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"]
Sep 09 14:32:37.278526 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:37.278505    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5b36371b/e2e-lifecycle-model-5b36371b-kserve-5654f69d5f-ftg8l"]
Sep 09 14:32:38.734973 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:32:38.734940    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac3ff2b-496a-45ab-a790-fd96875774e6" path="/var/lib/kubelet/pods/aac3ff2b-496a-45ab-a790-fd96875774e6/volumes"
Sep 09 14:33:42.478547 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.478464    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"]
Sep 09 14:33:42.479051 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.479036    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac3ff2b-496a-45ab-a790-fd96875774e6" containerName="main"
Sep 09 14:33:42.479051 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.479053    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac3ff2b-496a-45ab-a790-fd96875774e6" containerName="main"
Sep 09 14:33:42.479196 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.479068    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbd17d5-dd82-436a-9e7c-41588d2eccf8" containerName="maas-api"
Sep 09 14:33:42.479196 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.479074    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbd17d5-dd82-436a-9e7c-41588d2eccf8" containerName="maas-api"
Sep 09 14:33:42.479196 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.479176    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbd17d5-dd82-436a-9e7c-41588d2eccf8" containerName="maas-api"
Sep 09 14:33:42.479196 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.479184    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac3ff2b-496a-45ab-a790-fd96875774e6" containerName="main"
Sep 09 14:33:42.482298 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.482275    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.484831 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.484808    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e565aa\"/\"default-dockercfg-lxkht\""
Sep 09 14:33:42.484948 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.484858    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e565aa\"/\"kube-root-ca.crt\""
Sep 09 14:33:42.485755 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.485738    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e565aa\"/\"sub-test-model-e565aa-kserve-self-signed-certs\""
Sep 09 14:33:42.485851 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.485786    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e565aa\"/\"openshift-service-ca.crt\""
Sep 09 14:33:42.493629 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.493608    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"]
Sep 09 14:33:42.648000 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.647972    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5vf5\" (UniqueName: \"kubernetes.io/projected/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-kube-api-access-g5vf5\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.648149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.648039    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-model-cache\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.648149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.648135    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tmp-dir\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.648235 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.648162    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-home\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.648271 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.648234    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-dshm\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.648271 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.648255    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tls-certs\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.748605 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.748581    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-dshm\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.748724 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.748616    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tls-certs\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.748724 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.748644    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5vf5\" (UniqueName: \"kubernetes.io/projected/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-kube-api-access-g5vf5\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.748724 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.748699    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-model-cache\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.748883 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.748745    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tmp-dir\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.748883 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.748767    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-home\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.749078 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.749048    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-model-cache\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.749217 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.749128    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tmp-dir\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.749217 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.749208    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-home\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.750831 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.750813    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-dshm\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.751113 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.751082    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tls-certs\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.756388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.756367    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5vf5\" (UniqueName: \"kubernetes.io/projected/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-kube-api-access-g5vf5\") pod \"sub-test-model-e565aa-kserve-74685c647c-rvt58\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:42.793174 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:42.793152    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:43.062392 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:43.062366    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"]
Sep 09 14:33:43.063773 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:33:43.063742    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d625fdc_bb46_44e5_af21_b171bc5c4a5d.slice/crio-29b664ec2d66add1cbd8070b35f49557b076274642167a67742aed8f9cb84e62 WatchSource:0}: Error finding container 29b664ec2d66add1cbd8070b35f49557b076274642167a67742aed8f9cb84e62: Status 404 returned error can't find the container with id 29b664ec2d66add1cbd8070b35f49557b076274642167a67742aed8f9cb84e62
Sep 09 14:33:43.594978 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:43.594946    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58" event={"ID":"1d625fdc-bb46-44e5-af21-b171bc5c4a5d","Type":"ContainerStarted","Data":"079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0"}
Sep 09 14:33:43.595402 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:43.594985    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58" event={"ID":"1d625fdc-bb46-44e5-af21-b171bc5c4a5d","Type":"ContainerStarted","Data":"29b664ec2d66add1cbd8070b35f49557b076274642167a67742aed8f9cb84e62"}
Sep 09 14:33:43.614366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:43.614318    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58" podStartSLOduration=1.614305543 podStartE2EDuration="1.614305543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:43.61129258 +0000 UTC m=+1357.374087821" watchObservedRunningTime="2026-09-09 14:33:43.614305543 +0000 UTC m=+1357.377100780"
Sep 09 14:33:52.794191 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:52.794156    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:52.794730 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:52.794210    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:52.806826 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:52.806795    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:33:53.656241 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:33:53.656208    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:34:05.813319 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.813283    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"]
Sep 09 14:34:05.817227 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.817205    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:34:05.820086 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.820064    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4b83c2\""
Sep 09 14:34:05.820221 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.820064    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4b83c2-dockercfg-rqwqj\""
Sep 09 14:34:05.827998 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.827974    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"]
Sep 09 14:34:05.853856 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.853826    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knnfz\" (UniqueName: \"kubernetes.io/projected/52eabd76-07df-46a5-a357-db33b0fcd042-kube-api-access-knnfz\") pod \"payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4\" (UID: \"52eabd76-07df-46a5-a357-db33b0fcd042\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:34:05.854009 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.853975    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52eabd76-07df-46a5-a357-db33b0fcd042-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4\" (UID: \"52eabd76-07df-46a5-a357-db33b0fcd042\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:34:05.955188 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.955161    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52eabd76-07df-46a5-a357-db33b0fcd042-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4\" (UID: \"52eabd76-07df-46a5-a357-db33b0fcd042\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:34:05.955305 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.955197    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knnfz\" (UniqueName: \"kubernetes.io/projected/52eabd76-07df-46a5-a357-db33b0fcd042-kube-api-access-knnfz\") pod \"payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4\" (UID: \"52eabd76-07df-46a5-a357-db33b0fcd042\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:34:05.955797 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.955780    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52eabd76-07df-46a5-a357-db33b0fcd042-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4\" (UID: \"52eabd76-07df-46a5-a357-db33b0fcd042\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:34:05.963194 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:05.963173    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knnfz\" (UniqueName: \"kubernetes.io/projected/52eabd76-07df-46a5-a357-db33b0fcd042-kube-api-access-knnfz\") pod \"payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4\" (UID: \"52eabd76-07df-46a5-a357-db33b0fcd042\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:34:06.130046 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:06.129969    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:34:06.420293 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:06.420266    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"]
Sep 09 14:34:06.423130 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:34:06.423047    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52eabd76_07df_46a5_a357_db33b0fcd042.slice/crio-11d78c1820784292acea921eb3ce1c90216cbe369cbc2f4b8e077ea0d4f7dbc0 WatchSource:0}: Error finding container 11d78c1820784292acea921eb3ce1c90216cbe369cbc2f4b8e077ea0d4f7dbc0: Status 404 returned error can't find the container with id 11d78c1820784292acea921eb3ce1c90216cbe369cbc2f4b8e077ea0d4f7dbc0
Sep 09 14:34:06.722140 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:06.722047    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4" event={"ID":"52eabd76-07df-46a5-a357-db33b0fcd042","Type":"ContainerStarted","Data":"f246421f2359295392e27a5879bdb650d730055c29c7acd86c3081b40a26648d"}
Sep 09 14:34:06.722140 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:06.722124    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4" event={"ID":"52eabd76-07df-46a5-a357-db33b0fcd042","Type":"ContainerStarted","Data":"11d78c1820784292acea921eb3ce1c90216cbe369cbc2f4b8e077ea0d4f7dbc0"}
Sep 09 14:34:06.722329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:06.722243    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:34:06.740307 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:06.740009    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4" podStartSLOduration=1.7399912020000001 podStartE2EDuration="1.739991202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:34:06.739597802 +0000 UTC m=+1380.502393043" watchObservedRunningTime="2026-09-09 14:34:06.739991202 +0000 UTC m=+1380.502786441"
Sep 09 14:34:17.728969 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:17.728931    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:34:34.486340 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.486305    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"]
Sep 09 14:34:34.492425 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.492400    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.494602 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.494580    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a959bc-serving-cert\""
Sep 09 14:34:34.502069 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.501834    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"]
Sep 09 14:34:34.602515 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.602483    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-maas-api-tls\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.602656 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.602594    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321e95db-5986-4a13-a583-80969cdc82a3-openshift-service-ca\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.602656 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.602628    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7hb\" (UniqueName: \"kubernetes.io/projected/321e95db-5986-4a13-a583-80969cdc82a3-kube-api-access-rt7hb\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.602740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.602703    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-metrics-certs\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.703987 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.703960    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-metrics-certs\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.704144 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.703997    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-maas-api-tls\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.704144 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.704051    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321e95db-5986-4a13-a583-80969cdc82a3-openshift-service-ca\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.704144 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.704068    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7hb\" (UniqueName: \"kubernetes.io/projected/321e95db-5986-4a13-a583-80969cdc82a3-kube-api-access-rt7hb\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.704342 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:34:34.704181    2532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-serving-cert: secret "maas-api-e2e-api-b-a959bc-serving-cert" not found
Sep 09 14:34:34.704342 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:34:34.704238    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-maas-api-tls podName:321e95db-5986-4a13-a583-80969cdc82a3 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:35.204220661 +0000 UTC m=+1408.967015880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-maas-api-tls") pod "maas-api-e2e-api-b-a959bc-58654585cb-d4csj" (UID: "321e95db-5986-4a13-a583-80969cdc82a3") : secret "maas-api-e2e-api-b-a959bc-serving-cert" not found
Sep 09 14:34:34.704720 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.704700    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321e95db-5986-4a13-a583-80969cdc82a3-openshift-service-ca\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.706392 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.706367    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-metrics-certs\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:34.712669 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:34.712649    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7hb\" (UniqueName: \"kubernetes.io/projected/321e95db-5986-4a13-a583-80969cdc82a3-kube-api-access-rt7hb\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:35.209752 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:35.209720    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-maas-api-tls\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:35.212070 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:35.212051    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-maas-api-tls\") pod \"maas-api-e2e-api-b-a959bc-58654585cb-d4csj\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:35.406069 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:35.406035    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:35.681172 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:35.681140    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"]
Sep 09 14:34:35.682886 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:34:35.682861    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321e95db_5986_4a13_a583_80969cdc82a3.slice/crio-897c467c3ecba3382033fc08ace6741baaecdd8032c21765abbfd8bc22522c38 WatchSource:0}: Error finding container 897c467c3ecba3382033fc08ace6741baaecdd8032c21765abbfd8bc22522c38: Status 404 returned error can't find the container with id 897c467c3ecba3382033fc08ace6741baaecdd8032c21765abbfd8bc22522c38
Sep 09 14:34:35.684250 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:35.684232    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 14:34:35.871777 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:35.871745    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj" event={"ID":"321e95db-5986-4a13-a583-80969cdc82a3","Type":"ContainerStarted","Data":"897c467c3ecba3382033fc08ace6741baaecdd8032c21765abbfd8bc22522c38"}
Sep 09 14:34:36.878287 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:36.878254    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj" event={"ID":"321e95db-5986-4a13-a583-80969cdc82a3","Type":"ContainerStarted","Data":"326e238eccb9c029eca09d2c33dd7fbf2aec38458c539307af7f410b0a0893e1"}
Sep 09 14:34:36.878637 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:36.878382    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:34:36.895597 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:36.895555    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj" podStartSLOduration=2.530703104 podStartE2EDuration="2.895540815s" totalImagesPullingTime="364.837711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:34:36.894733258 +0000 UTC m=+1410.657528501" watchObservedRunningTime="2026-09-09 14:34:36.895540815 +0000 UTC m=+1410.658336056"
Sep 09 14:34:42.888502 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:34:42.888469    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:35:16.866635 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:16.866603    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"]
Sep 09 14:35:16.867217 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:16.866824    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4" podUID="52eabd76-07df-46a5-a357-db33b0fcd042" containerName="payload-processing" containerID="cri-o://f246421f2359295392e27a5879bdb650d730055c29c7acd86c3081b40a26648d" gracePeriod=30
Sep 09 14:35:17.091850 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:17.091818    2532 generic.go:356] "Generic (PLEG): container finished" podID="52eabd76-07df-46a5-a357-db33b0fcd042" containerID="f246421f2359295392e27a5879bdb650d730055c29c7acd86c3081b40a26648d" exitCode=0
Sep 09 14:35:17.091996 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:17.091888    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4" event={"ID":"52eabd76-07df-46a5-a357-db33b0fcd042","Type":"ContainerDied","Data":"f246421f2359295392e27a5879bdb650d730055c29c7acd86c3081b40a26648d"}
Sep 09 14:35:17.120281 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:17.120231    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:35:17.214846 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:17.214822    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knnfz\" (UniqueName: \"kubernetes.io/projected/52eabd76-07df-46a5-a357-db33b0fcd042-kube-api-access-knnfz\") pod \"52eabd76-07df-46a5-a357-db33b0fcd042\" (UID: \"52eabd76-07df-46a5-a357-db33b0fcd042\") "
Sep 09 14:35:17.214979 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:17.214962    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52eabd76-07df-46a5-a357-db33b0fcd042-plugins-config-volume\") pod \"52eabd76-07df-46a5-a357-db33b0fcd042\" (UID: \"52eabd76-07df-46a5-a357-db33b0fcd042\") "
Sep 09 14:35:17.215350 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:17.215316    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52eabd76-07df-46a5-a357-db33b0fcd042-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52eabd76-07df-46a5-a357-db33b0fcd042" (UID: "52eabd76-07df-46a5-a357-db33b0fcd042"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:35:17.216874 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:17.216844    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52eabd76-07df-46a5-a357-db33b0fcd042-kube-api-access-knnfz" (OuterVolumeSpecName: "kube-api-access-knnfz") pod "52eabd76-07df-46a5-a357-db33b0fcd042" (UID: "52eabd76-07df-46a5-a357-db33b0fcd042"). InnerVolumeSpecName "kube-api-access-knnfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:35:17.316047 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:17.316024    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52eabd76-07df-46a5-a357-db33b0fcd042-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:17.316047 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:17.316045    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knnfz\" (UniqueName: \"kubernetes.io/projected/52eabd76-07df-46a5-a357-db33b0fcd042-kube-api-access-knnfz\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:18.098295 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:18.098263    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"
Sep 09 14:35:18.098714 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:18.098310    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4" event={"ID":"52eabd76-07df-46a5-a357-db33b0fcd042","Type":"ContainerDied","Data":"11d78c1820784292acea921eb3ce1c90216cbe369cbc2f4b8e077ea0d4f7dbc0"}
Sep 09 14:35:18.098714 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:18.098364    2532 scope.go:117] "RemoveContainer" containerID="f246421f2359295392e27a5879bdb650d730055c29c7acd86c3081b40a26648d"
Sep 09 14:35:18.120517 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:18.120491    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"]
Sep 09 14:35:18.129238 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:18.129216    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b83c2-7c5d8f66bc-6j6b4"]
Sep 09 14:35:18.734548 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:18.734519    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52eabd76-07df-46a5-a357-db33b0fcd042" path="/var/lib/kubelet/pods/52eabd76-07df-46a5-a357-db33b0fcd042/volumes"
Sep 09 14:35:38.817672 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:38.817641    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"]
Sep 09 14:35:39.967854 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:39.967813    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"]
Sep 09 14:35:39.968334 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:39.968155    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj" podUID="321e95db-5986-4a13-a583-80969cdc82a3" containerName="maas-api" containerID="cri-o://326e238eccb9c029eca09d2c33dd7fbf2aec38458c539307af7f410b0a0893e1" gracePeriod=30
Sep 09 14:35:40.215942 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.215912    2532 generic.go:356] "Generic (PLEG): container finished" podID="321e95db-5986-4a13-a583-80969cdc82a3" containerID="326e238eccb9c029eca09d2c33dd7fbf2aec38458c539307af7f410b0a0893e1" exitCode=0
Sep 09 14:35:40.216085 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.215964    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj" event={"ID":"321e95db-5986-4a13-a583-80969cdc82a3","Type":"ContainerDied","Data":"326e238eccb9c029eca09d2c33dd7fbf2aec38458c539307af7f410b0a0893e1"}
Sep 09 14:35:40.216085 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.216010    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj" event={"ID":"321e95db-5986-4a13-a583-80969cdc82a3","Type":"ContainerDied","Data":"897c467c3ecba3382033fc08ace6741baaecdd8032c21765abbfd8bc22522c38"}
Sep 09 14:35:40.216085 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.216033    2532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="897c467c3ecba3382033fc08ace6741baaecdd8032c21765abbfd8bc22522c38"
Sep 09 14:35:40.218399 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.218345    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:35:40.314319 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.314297    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321e95db-5986-4a13-a583-80969cdc82a3-openshift-service-ca\") pod \"321e95db-5986-4a13-a583-80969cdc82a3\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") "
Sep 09 14:35:40.314432 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.314384    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-metrics-certs\") pod \"321e95db-5986-4a13-a583-80969cdc82a3\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") "
Sep 09 14:35:40.314474 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.314441    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-maas-api-tls\") pod \"321e95db-5986-4a13-a583-80969cdc82a3\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") "
Sep 09 14:35:40.314513 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.314476    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt7hb\" (UniqueName: \"kubernetes.io/projected/321e95db-5986-4a13-a583-80969cdc82a3-kube-api-access-rt7hb\") pod \"321e95db-5986-4a13-a583-80969cdc82a3\" (UID: \"321e95db-5986-4a13-a583-80969cdc82a3\") "
Sep 09 14:35:40.314649 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.314625    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321e95db-5986-4a13-a583-80969cdc82a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "321e95db-5986-4a13-a583-80969cdc82a3" (UID: "321e95db-5986-4a13-a583-80969cdc82a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:35:40.314729 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.314714    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321e95db-5986-4a13-a583-80969cdc82a3-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:40.316572 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.316541    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "321e95db-5986-4a13-a583-80969cdc82a3" (UID: "321e95db-5986-4a13-a583-80969cdc82a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:35:40.316664 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.316577    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321e95db-5986-4a13-a583-80969cdc82a3-kube-api-access-rt7hb" (OuterVolumeSpecName: "kube-api-access-rt7hb") pod "321e95db-5986-4a13-a583-80969cdc82a3" (UID: "321e95db-5986-4a13-a583-80969cdc82a3"). InnerVolumeSpecName "kube-api-access-rt7hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:35:40.316664 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.316588    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "321e95db-5986-4a13-a583-80969cdc82a3" (UID: "321e95db-5986-4a13-a583-80969cdc82a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:35:40.415310 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.415279    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:40.415310 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.415300    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321e95db-5986-4a13-a583-80969cdc82a3-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:40.415310 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:40.415309    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt7hb\" (UniqueName: \"kubernetes.io/projected/321e95db-5986-4a13-a583-80969cdc82a3-kube-api-access-rt7hb\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:41.222117 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:41.222013    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"
Sep 09 14:35:41.239843 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:41.239813    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"]
Sep 09 14:35:41.243695 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:41.243673    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a959bc-58654585cb-d4csj"]
Sep 09 14:35:42.734922 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:42.734878    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="321e95db-5986-4a13-a583-80969cdc82a3" path="/var/lib/kubelet/pods/321e95db-5986-4a13-a583-80969cdc82a3/volumes"
Sep 09 14:35:47.698828 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.698795    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"]
Sep 09 14:35:47.699284 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.699270    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52eabd76-07df-46a5-a357-db33b0fcd042" containerName="payload-processing"
Sep 09 14:35:47.699331 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.699286    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="52eabd76-07df-46a5-a357-db33b0fcd042" containerName="payload-processing"
Sep 09 14:35:47.699331 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.699310    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="321e95db-5986-4a13-a583-80969cdc82a3" containerName="maas-api"
Sep 09 14:35:47.699331 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.699315    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="321e95db-5986-4a13-a583-80969cdc82a3" containerName="maas-api"
Sep 09 14:35:47.699432 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.699380    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="52eabd76-07df-46a5-a357-db33b0fcd042" containerName="payload-processing"
Sep 09 14:35:47.699432 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.699390    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="321e95db-5986-4a13-a583-80969cdc82a3" containerName="maas-api"
Sep 09 14:35:47.702715 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.702698    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.706069 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.705465    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e565aa\"/\"rate-test-model-e565aa-kserve-self-signed-certs\""
Sep 09 14:35:47.712832 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.712809    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"]
Sep 09 14:35:47.781258 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.781237    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-home\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.781390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.781262    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-dshm\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.781390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.781304    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tls-certs\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.781390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.781331    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tmp-dir\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.781390 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.781354    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ml6p\" (UniqueName: \"kubernetes.io/projected/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-kube-api-access-6ml6p\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.781548 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.781417    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-model-cache\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.882076 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.882048    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ml6p\" (UniqueName: \"kubernetes.io/projected/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-kube-api-access-6ml6p\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.882212 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.882087    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-model-cache\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.882212 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.882177    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-home\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.882313 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.882292    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-dshm\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.882407 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.882389    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tls-certs\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.882463 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.882449    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tmp-dir\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.882599 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.882581    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-model-cache\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.882639 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.882609    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-home\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.882731 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.882708    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tmp-dir\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.884308 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.884288    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-dshm\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.884840 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.884819    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tls-certs\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:47.889906 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:47.889879    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ml6p\" (UniqueName: \"kubernetes.io/projected/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-kube-api-access-6ml6p\") pod \"rate-test-model-e565aa-kserve-7c564854c9-hxbpk\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") " pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:48.015057 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:48.015032    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:48.286361 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:48.286335    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"]
Sep 09 14:35:48.287818 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:35:48.287791    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50c0bb7e_aaca_4e52_a5ee_42c7c325fd63.slice/crio-0a156d51974382fc2ff9436985bc06e5c7e9b499a0ecd3038a95438e749b0d82 WatchSource:0}: Error finding container 0a156d51974382fc2ff9436985bc06e5c7e9b499a0ecd3038a95438e749b0d82: Status 404 returned error can't find the container with id 0a156d51974382fc2ff9436985bc06e5c7e9b499a0ecd3038a95438e749b0d82
Sep 09 14:35:49.266514 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:49.266482    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk" event={"ID":"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63","Type":"ContainerStarted","Data":"3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2"}
Sep 09 14:35:49.266884 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:49.266524    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk" event={"ID":"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63","Type":"ContainerStarted","Data":"0a156d51974382fc2ff9436985bc06e5c7e9b499a0ecd3038a95438e749b0d82"}
Sep 09 14:35:49.283703 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:49.283645    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk" podStartSLOduration=2.283630575 podStartE2EDuration="2.283630575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:35:49.282164175 +0000 UTC m=+1483.044959414" watchObservedRunningTime="2026-09-09 14:35:49.283630575 +0000 UTC m=+1483.046425815"
Sep 09 14:35:53.837305 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:53.837252    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58" podUID="1d625fdc-bb46-44e5-af21-b171bc5c4a5d" containerName="main" containerID="cri-o://079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0" gracePeriod=45
Sep 09 14:35:54.082316 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.082290    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:35:54.240159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240136    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5vf5\" (UniqueName: \"kubernetes.io/projected/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-kube-api-access-g5vf5\") pod \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") "
Sep 09 14:35:54.240283 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240170    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tls-certs\") pod \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") "
Sep 09 14:35:54.240283 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240241    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-dshm\") pod \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") "
Sep 09 14:35:54.240283 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240265    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tmp-dir\") pod \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") "
Sep 09 14:35:54.240454 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240301    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-model-cache\") pod \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") "
Sep 09 14:35:54.240454 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240345    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-home\") pod \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\" (UID: \"1d625fdc-bb46-44e5-af21-b171bc5c4a5d\") "
Sep 09 14:35:54.240560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240538    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d625fdc-bb46-44e5-af21-b171bc5c4a5d" (UID: "1d625fdc-bb46-44e5-af21-b171bc5c4a5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:35:54.240623 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240591    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d625fdc-bb46-44e5-af21-b171bc5c4a5d" (UID: "1d625fdc-bb46-44e5-af21-b171bc5c4a5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:35:54.240673 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240648    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-home" (OuterVolumeSpecName: "home") pod "1d625fdc-bb46-44e5-af21-b171bc5c4a5d" (UID: "1d625fdc-bb46-44e5-af21-b171bc5c4a5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:35:54.240723 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240704    2532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tmp-dir\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:54.240767 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.240730    2532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-model-cache\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:54.242314 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.242295    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-kube-api-access-g5vf5" (OuterVolumeSpecName: "kube-api-access-g5vf5") pod "1d625fdc-bb46-44e5-af21-b171bc5c4a5d" (UID: "1d625fdc-bb46-44e5-af21-b171bc5c4a5d"). InnerVolumeSpecName "kube-api-access-g5vf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:35:54.242697 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.242681    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-dshm" (OuterVolumeSpecName: "dshm") pod "1d625fdc-bb46-44e5-af21-b171bc5c4a5d" (UID: "1d625fdc-bb46-44e5-af21-b171bc5c4a5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:35:54.242766 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.242694    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d625fdc-bb46-44e5-af21-b171bc5c4a5d" (UID: "1d625fdc-bb46-44e5-af21-b171bc5c4a5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:35:54.293146 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.293117    2532 generic.go:356] "Generic (PLEG): container finished" podID="1d625fdc-bb46-44e5-af21-b171bc5c4a5d" containerID="079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0" exitCode=0
Sep 09 14:35:54.293252 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.293158    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"
Sep 09 14:35:54.293252 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.293156    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58" event={"ID":"1d625fdc-bb46-44e5-af21-b171bc5c4a5d","Type":"ContainerDied","Data":"079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0"}
Sep 09 14:35:54.293252 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.293212    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58" event={"ID":"1d625fdc-bb46-44e5-af21-b171bc5c4a5d","Type":"ContainerDied","Data":"29b664ec2d66add1cbd8070b35f49557b076274642167a67742aed8f9cb84e62"}
Sep 09 14:35:54.293252 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.293236    2532 scope.go:117] "RemoveContainer" containerID="079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0"
Sep 09 14:35:54.302442 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.302424    2532 scope.go:117] "RemoveContainer" containerID="079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0"
Sep 09 14:35:54.302679 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:35:54.302660    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0\": container with ID starting with 079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0 not found: ID does not exist" containerID="079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0"
Sep 09 14:35:54.302732 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.302687    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0"} err="failed to get container status \"079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0\": rpc error: code = NotFound desc = could not find container \"079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0\": container with ID starting with 079148795d516839b1c26822b390752c76a20d806a77fad1bc31c0b7e21273f0 not found: ID does not exist"
Sep 09 14:35:54.313575 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.313550    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"]
Sep 09 14:35:54.317122 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.317089    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/sub-test-model-e565aa-kserve-74685c647c-rvt58"]
Sep 09 14:35:54.342224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.342205    2532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-home\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:54.342224 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.342223    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5vf5\" (UniqueName: \"kubernetes.io/projected/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-kube-api-access-g5vf5\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:54.342330 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.342233    2532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-tls-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:54.342330 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.342242    2532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d625fdc-bb46-44e5-af21-b171bc5c4a5d-dshm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:54.735066 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:54.735039    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d625fdc-bb46-44e5-af21-b171bc5c4a5d" path="/var/lib/kubelet/pods/1d625fdc-bb46-44e5-af21-b171bc5c4a5d/volumes"
Sep 09 14:35:57.637976 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:57.637946    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"]
Sep 09 14:35:57.638392 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:57.638138    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj" podUID="5f273552-4024-4df1-a8d9-a068edf2c89b" containerName="maas-api" containerID="cri-o://add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440" gracePeriod=30
Sep 09 14:35:57.905162 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:57.905138    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:35:58.015269 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.015241    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:58.015509 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.015489    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:58.027524 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.027502    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:58.074870 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.074847    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8v76\" (UniqueName: \"kubernetes.io/projected/5f273552-4024-4df1-a8d9-a068edf2c89b-kube-api-access-w8v76\") pod \"5f273552-4024-4df1-a8d9-a068edf2c89b\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") "
Sep 09 14:35:58.074978 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.074964    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-maas-api-tls\") pod \"5f273552-4024-4df1-a8d9-a068edf2c89b\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") "
Sep 09 14:35:58.075031 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.074993    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f273552-4024-4df1-a8d9-a068edf2c89b-openshift-service-ca\") pod \"5f273552-4024-4df1-a8d9-a068edf2c89b\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") "
Sep 09 14:35:58.075031 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.075012    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-metrics-certs\") pod \"5f273552-4024-4df1-a8d9-a068edf2c89b\" (UID: \"5f273552-4024-4df1-a8d9-a068edf2c89b\") "
Sep 09 14:35:58.075403 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.075378    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f273552-4024-4df1-a8d9-a068edf2c89b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f273552-4024-4df1-a8d9-a068edf2c89b" (UID: "5f273552-4024-4df1-a8d9-a068edf2c89b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:35:58.076874 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.076851    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f273552-4024-4df1-a8d9-a068edf2c89b-kube-api-access-w8v76" (OuterVolumeSpecName: "kube-api-access-w8v76") pod "5f273552-4024-4df1-a8d9-a068edf2c89b" (UID: "5f273552-4024-4df1-a8d9-a068edf2c89b"). InnerVolumeSpecName "kube-api-access-w8v76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:35:58.077027 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.077011    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f273552-4024-4df1-a8d9-a068edf2c89b" (UID: "5f273552-4024-4df1-a8d9-a068edf2c89b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:35:58.077084 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.077065    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5f273552-4024-4df1-a8d9-a068edf2c89b" (UID: "5f273552-4024-4df1-a8d9-a068edf2c89b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:35:58.176463 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.176410    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:58.176463 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.176434    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f273552-4024-4df1-a8d9-a068edf2c89b-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:58.176463 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.176445    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f273552-4024-4df1-a8d9-a068edf2c89b-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:58.176463 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.176453    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8v76\" (UniqueName: \"kubernetes.io/projected/5f273552-4024-4df1-a8d9-a068edf2c89b-kube-api-access-w8v76\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:35:58.314340 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.314310    2532 generic.go:356] "Generic (PLEG): container finished" podID="5f273552-4024-4df1-a8d9-a068edf2c89b" containerID="add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440" exitCode=0
Sep 09 14:35:58.314440 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.314354    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"
Sep 09 14:35:58.314440 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.314399    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj" event={"ID":"5f273552-4024-4df1-a8d9-a068edf2c89b","Type":"ContainerDied","Data":"add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440"}
Sep 09 14:35:58.314521 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.314450    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj" event={"ID":"5f273552-4024-4df1-a8d9-a068edf2c89b","Type":"ContainerDied","Data":"e3fde47768d644dea7e7161e893167a0dfc79981bd44a384bfa9aa28893d60ca"}
Sep 09 14:35:58.314521 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.314479    2532 scope.go:117] "RemoveContainer" containerID="add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440"
Sep 09 14:35:58.325297 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.325280    2532 scope.go:117] "RemoveContainer" containerID="add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440"
Sep 09 14:35:58.325561 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:35:58.325541    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440\": container with ID starting with add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440 not found: ID does not exist" containerID="add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440"
Sep 09 14:35:58.325634 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.325572    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440"} err="failed to get container status \"add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440\": rpc error: code = NotFound desc = could not find container \"add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440\": container with ID starting with add878d129b45c86eca7e86544574ad4381704150c97b51d520cbc6b5abdc440 not found: ID does not exist"
Sep 09 14:35:58.328309 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.328292    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:35:58.335529 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.335509    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"]
Sep 09 14:35:58.338621 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.338599    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb7fb-7967f85bf5-h8qbj"]
Sep 09 14:35:58.734898 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:35:58.734866    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f273552-4024-4df1-a8d9-a068edf2c89b" path="/var/lib/kubelet/pods/5f273552-4024-4df1-a8d9-a068edf2c89b/volumes"
Sep 09 14:36:06.775789 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:06.775760    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:36:06.777619 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:06.777600    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:36:15.419868 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.419835    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"]
Sep 09 14:36:15.420439 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.420028    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8" podUID="9a9e8981-c92a-4858-98e5-396f3bb05538" containerName="maas-api" containerID="cri-o://161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2" gracePeriod=30
Sep 09 14:36:15.473990 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.473961    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"]
Sep 09 14:36:15.474238 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.474186    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7" podUID="84c9851c-c9c1-476d-86e1-643f04f2c4f2" containerName="payload-processing" containerID="cri-o://e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e" gracePeriod=30
Sep 09 14:36:15.743895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.743871    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:36:15.746762 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.746738    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:36:15.828143 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.828114    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c9851c-c9c1-476d-86e1-643f04f2c4f2-plugins-config-volume\") pod \"84c9851c-c9c1-476d-86e1-643f04f2c4f2\" (UID: \"84c9851c-c9c1-476d-86e1-643f04f2c4f2\") "
Sep 09 14:36:15.828308 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.828163    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk8lx\" (UniqueName: \"kubernetes.io/projected/9a9e8981-c92a-4858-98e5-396f3bb05538-kube-api-access-sk8lx\") pod \"9a9e8981-c92a-4858-98e5-396f3bb05538\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") "
Sep 09 14:36:15.828308 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.828242    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-maas-api-tls\") pod \"9a9e8981-c92a-4858-98e5-396f3bb05538\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") "
Sep 09 14:36:15.828308 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.828275    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9e8981-c92a-4858-98e5-396f3bb05538-openshift-service-ca\") pod \"9a9e8981-c92a-4858-98e5-396f3bb05538\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") "
Sep 09 14:36:15.828485 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.828345    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2vhh\" (UniqueName: \"kubernetes.io/projected/84c9851c-c9c1-476d-86e1-643f04f2c4f2-kube-api-access-r2vhh\") pod \"84c9851c-c9c1-476d-86e1-643f04f2c4f2\" (UID: \"84c9851c-c9c1-476d-86e1-643f04f2c4f2\") "
Sep 09 14:36:15.828485 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.828377    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-metrics-certs\") pod \"9a9e8981-c92a-4858-98e5-396f3bb05538\" (UID: \"9a9e8981-c92a-4858-98e5-396f3bb05538\") "
Sep 09 14:36:15.828598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.828509    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c9851c-c9c1-476d-86e1-643f04f2c4f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84c9851c-c9c1-476d-86e1-643f04f2c4f2" (UID: "84c9851c-c9c1-476d-86e1-643f04f2c4f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:36:15.828752 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.828720    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9e8981-c92a-4858-98e5-396f3bb05538-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a9e8981-c92a-4858-98e5-396f3bb05538" (UID: "9a9e8981-c92a-4858-98e5-396f3bb05538"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:36:15.828752 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.828730    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c9851c-c9c1-476d-86e1-643f04f2c4f2-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:36:15.830324 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.830296    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9a9e8981-c92a-4858-98e5-396f3bb05538" (UID: "9a9e8981-c92a-4858-98e5-396f3bb05538"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:36:15.830458 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.830439    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9e8981-c92a-4858-98e5-396f3bb05538-kube-api-access-sk8lx" (OuterVolumeSpecName: "kube-api-access-sk8lx") pod "9a9e8981-c92a-4858-98e5-396f3bb05538" (UID: "9a9e8981-c92a-4858-98e5-396f3bb05538"). InnerVolumeSpecName "kube-api-access-sk8lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:36:15.830740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.830716    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a9e8981-c92a-4858-98e5-396f3bb05538" (UID: "9a9e8981-c92a-4858-98e5-396f3bb05538"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:36:15.830740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.830727    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c9851c-c9c1-476d-86e1-643f04f2c4f2-kube-api-access-r2vhh" (OuterVolumeSpecName: "kube-api-access-r2vhh") pod "84c9851c-c9c1-476d-86e1-643f04f2c4f2" (UID: "84c9851c-c9c1-476d-86e1-643f04f2c4f2"). InnerVolumeSpecName "kube-api-access-r2vhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:36:15.929844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.929781    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2vhh\" (UniqueName: \"kubernetes.io/projected/84c9851c-c9c1-476d-86e1-643f04f2c4f2-kube-api-access-r2vhh\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:36:15.929844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.929814    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:36:15.929844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.929829    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk8lx\" (UniqueName: \"kubernetes.io/projected/9a9e8981-c92a-4858-98e5-396f3bb05538-kube-api-access-sk8lx\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:36:15.929844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.929843    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9e8981-c92a-4858-98e5-396f3bb05538-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:36:15.930040 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:15.929857    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9e8981-c92a-4858-98e5-396f3bb05538-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:36:16.419030 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.418998    2532 generic.go:356] "Generic (PLEG): container finished" podID="84c9851c-c9c1-476d-86e1-643f04f2c4f2" containerID="e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e" exitCode=0
Sep 09 14:36:16.419234 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.419036    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"
Sep 09 14:36:16.419234 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.419073    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7" event={"ID":"84c9851c-c9c1-476d-86e1-643f04f2c4f2","Type":"ContainerDied","Data":"e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e"}
Sep 09 14:36:16.419234 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.419137    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7" event={"ID":"84c9851c-c9c1-476d-86e1-643f04f2c4f2","Type":"ContainerDied","Data":"b9a495a5f466631069f4e7495a17a49943148f867e037ca06f46b9e4acb6be8a"}
Sep 09 14:36:16.419234 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.419152    2532 scope.go:117] "RemoveContainer" containerID="e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e"
Sep 09 14:36:16.420418 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.420397    2532 generic.go:356] "Generic (PLEG): container finished" podID="9a9e8981-c92a-4858-98e5-396f3bb05538" containerID="161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2" exitCode=0
Sep 09 14:36:16.420754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.420449    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"
Sep 09 14:36:16.420754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.420462    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8" event={"ID":"9a9e8981-c92a-4858-98e5-396f3bb05538","Type":"ContainerDied","Data":"161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2"}
Sep 09 14:36:16.420754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.420506    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8" event={"ID":"9a9e8981-c92a-4858-98e5-396f3bb05538","Type":"ContainerDied","Data":"a8a980928259898fb097a099cde3d0bdd1a9e83f064e639bda095eb279ce186c"}
Sep 09 14:36:16.430179 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.429995    2532 scope.go:117] "RemoveContainer" containerID="e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e"
Sep 09 14:36:16.430294 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:36:16.430275    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e\": container with ID starting with e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e not found: ID does not exist" containerID="e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e"
Sep 09 14:36:16.430344 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.430303    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e"} err="failed to get container status \"e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e\": rpc error: code = NotFound desc = could not find container \"e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e\": container with ID starting with e57396d30dad4578e8b2a16f4cc5e8cd91a2abe6c7be7b47973ecaf2acba637e not found: ID does not exist"
Sep 09 14:36:16.430344 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.430320    2532 scope.go:117] "RemoveContainer" containerID="161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2"
Sep 09 14:36:16.443982 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.443967    2532 scope.go:117] "RemoveContainer" containerID="161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2"
Sep 09 14:36:16.444266 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:36:16.444249    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2\": container with ID starting with 161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2 not found: ID does not exist" containerID="161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2"
Sep 09 14:36:16.444321 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.444271    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2"} err="failed to get container status \"161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2\": rpc error: code = NotFound desc = could not find container \"161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2\": container with ID starting with 161c9b653bdeab8092a6875e25da5eac7c02632c0624d39e0523b8f66a1c3fc2 not found: ID does not exist"
Sep 09 14:36:16.453947 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.453925    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"]
Sep 09 14:36:16.462548 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.462529    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fecaf9-54d957f4ff-7z5s7"]
Sep 09 14:36:16.479085 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.479060    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"]
Sep 09 14:36:16.482761 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.482741    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fecaf9-748fd94958-wcbh8"]
Sep 09 14:36:16.734571 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.734544    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84c9851c-c9c1-476d-86e1-643f04f2c4f2" path="/var/lib/kubelet/pods/84c9851c-c9c1-476d-86e1-643f04f2c4f2/volumes"
Sep 09 14:36:16.734931 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:16.734918    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a9e8981-c92a-4858-98e5-396f3bb05538" path="/var/lib/kubelet/pods/9a9e8981-c92a-4858-98e5-396f3bb05538/volumes"
Sep 09 14:36:48.631668 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:48.631582    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"]
Sep 09 14:36:54.416597 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.416560    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"]
Sep 09 14:36:54.416978 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.416959    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a9e8981-c92a-4858-98e5-396f3bb05538" containerName="maas-api"
Sep 09 14:36:54.416978 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.416970    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a9e8981-c92a-4858-98e5-396f3bb05538" containerName="maas-api"
Sep 09 14:36:54.417063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.417001    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f273552-4024-4df1-a8d9-a068edf2c89b" containerName="maas-api"
Sep 09 14:36:54.417063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.417009    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f273552-4024-4df1-a8d9-a068edf2c89b" containerName="maas-api"
Sep 09 14:36:54.417063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.417019    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c9851c-c9c1-476d-86e1-643f04f2c4f2" containerName="payload-processing"
Sep 09 14:36:54.417063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.417028    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c9851c-c9c1-476d-86e1-643f04f2c4f2" containerName="payload-processing"
Sep 09 14:36:54.417063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.417047    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d625fdc-bb46-44e5-af21-b171bc5c4a5d" containerName="main"
Sep 09 14:36:54.417063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.417057    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d625fdc-bb46-44e5-af21-b171bc5c4a5d" containerName="main"
Sep 09 14:36:54.417299 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.417194    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a9e8981-c92a-4858-98e5-396f3bb05538" containerName="maas-api"
Sep 09 14:36:54.417299 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.417204    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f273552-4024-4df1-a8d9-a068edf2c89b" containerName="maas-api"
Sep 09 14:36:54.417299 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.417211    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d625fdc-bb46-44e5-af21-b171bc5c4a5d" containerName="main"
Sep 09 14:36:54.417299 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.417220    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c9851c-c9c1-476d-86e1-643f04f2c4f2" containerName="payload-processing"
Sep 09 14:36:54.420384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.420367    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.422775 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.422749    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ea13e2-serving-cert\""
Sep 09 14:36:54.429745 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.429722    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"]
Sep 09 14:36:54.474743 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.474714    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-metrics-certs\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.474894 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.474750    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lcg2\" (UniqueName: \"kubernetes.io/projected/a483f9a3-6b69-4838-8e4c-96916f1a5546-kube-api-access-7lcg2\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.474894 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.474827    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.474894 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.474891    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a483f9a3-6b69-4838-8e4c-96916f1a5546-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.575374 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.575350    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.575484 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.575403    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a483f9a3-6b69-4838-8e4c-96916f1a5546-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.575484 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.575475    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-metrics-certs\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.575593 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.575505    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lcg2\" (UniqueName: \"kubernetes.io/projected/a483f9a3-6b69-4838-8e4c-96916f1a5546-kube-api-access-7lcg2\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.576144 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.576123    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a483f9a3-6b69-4838-8e4c-96916f1a5546-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.577717 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.577697    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-metrics-certs\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.577804 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.577787    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.582836 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.582812    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lcg2\" (UniqueName: \"kubernetes.io/projected/a483f9a3-6b69-4838-8e4c-96916f1a5546-kube-api-access-7lcg2\") pod \"maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:54.733379 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:54.733349    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:55.004603 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.004572    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"]
Sep 09 14:36:55.006607 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:36:55.006579    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda483f9a3_6b69_4838_8e4c_96916f1a5546.slice/crio-0976461d7ec2c3c456465ff4cfeaf1df2e947fc0ce983bc43f0858923758c42c WatchSource:0}: Error finding container 0976461d7ec2c3c456465ff4cfeaf1df2e947fc0ce983bc43f0858923758c42c: Status 404 returned error can't find the container with id 0976461d7ec2c3c456465ff4cfeaf1df2e947fc0ce983bc43f0858923758c42c
Sep 09 14:36:55.040385 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.040356    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"]
Sep 09 14:36:55.045394 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.045375    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:36:55.047755 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.047736    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ea13e2\""
Sep 09 14:36:55.047853 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.047808    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ea13e2-dockercfg-m7qvl\""
Sep 09 14:36:55.054363 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.054342    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"]
Sep 09 14:36:55.079205 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.079182    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4bzg\" (UniqueName: \"kubernetes.io/projected/1ffbb714-c838-4c10-accd-fdd35ed06a97-kube-api-access-j4bzg\") pod \"payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c\" (UID: \"1ffbb714-c838-4c10-accd-fdd35ed06a97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:36:55.079341 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.079324    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffbb714-c838-4c10-accd-fdd35ed06a97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c\" (UID: \"1ffbb714-c838-4c10-accd-fdd35ed06a97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:36:55.180431 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.180399    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4bzg\" (UniqueName: \"kubernetes.io/projected/1ffbb714-c838-4c10-accd-fdd35ed06a97-kube-api-access-j4bzg\") pod \"payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c\" (UID: \"1ffbb714-c838-4c10-accd-fdd35ed06a97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:36:55.180550 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.180491    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffbb714-c838-4c10-accd-fdd35ed06a97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c\" (UID: \"1ffbb714-c838-4c10-accd-fdd35ed06a97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:36:55.181161 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.181143    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffbb714-c838-4c10-accd-fdd35ed06a97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c\" (UID: \"1ffbb714-c838-4c10-accd-fdd35ed06a97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:36:55.191342 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.191318    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4bzg\" (UniqueName: \"kubernetes.io/projected/1ffbb714-c838-4c10-accd-fdd35ed06a97-kube-api-access-j4bzg\") pod \"payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c\" (UID: \"1ffbb714-c838-4c10-accd-fdd35ed06a97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:36:55.356932 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.356872    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:36:55.627123 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.626978    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592" event={"ID":"a483f9a3-6b69-4838-8e4c-96916f1a5546","Type":"ContainerStarted","Data":"0976461d7ec2c3c456465ff4cfeaf1df2e947fc0ce983bc43f0858923758c42c"}
Sep 09 14:36:55.628582 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:55.628560    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"]
Sep 09 14:36:55.629946 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:36:55.629917    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffbb714_c838_4c10_accd_fdd35ed06a97.slice/crio-7a858a7b8565268b0ea1055a8c47887c132137618ca654f43d01dd30d6ed3451 WatchSource:0}: Error finding container 7a858a7b8565268b0ea1055a8c47887c132137618ca654f43d01dd30d6ed3451: Status 404 returned error can't find the container with id 7a858a7b8565268b0ea1055a8c47887c132137618ca654f43d01dd30d6ed3451
Sep 09 14:36:56.633714 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:56.633681    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592" event={"ID":"a483f9a3-6b69-4838-8e4c-96916f1a5546","Type":"ContainerStarted","Data":"4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927"}
Sep 09 14:36:56.634150 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:56.633774    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:36:56.635761 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:56.635715    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c" event={"ID":"1ffbb714-c838-4c10-accd-fdd35ed06a97","Type":"ContainerStarted","Data":"3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6"}
Sep 09 14:36:56.635900 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:56.635796    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c" event={"ID":"1ffbb714-c838-4c10-accd-fdd35ed06a97","Type":"ContainerStarted","Data":"7a858a7b8565268b0ea1055a8c47887c132137618ca654f43d01dd30d6ed3451"}
Sep 09 14:36:56.635900 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:56.635839    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:36:56.654935 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:56.654887    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592" podStartSLOduration=1.811030688 podStartE2EDuration="2.654873793s" totalImagesPullingTime="843.843105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:36:56.652668477 +0000 UTC m=+1550.415463719" watchObservedRunningTime="2026-09-09 14:36:56.654873793 +0000 UTC m=+1550.417669033"
Sep 09 14:36:56.668993 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:36:56.668947    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c" podStartSLOduration=1.6689239919999999 podStartE2EDuration="1.668923992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:36:56.668028558 +0000 UTC m=+1550.430823798" watchObservedRunningTime="2026-09-09 14:36:56.668923992 +0000 UTC m=+1550.431719231"
Sep 09 14:37:02.647379 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:02.647350    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:37:03.651613 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.651545    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk" podUID="50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" containerName="main" containerID="cri-o://3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2" gracePeriod=45
Sep 09 14:37:03.899730 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.899696    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:37:03.958847 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.958776    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tmp-dir\") pod \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") "
Sep 09 14:37:03.958847 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.958816    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-dshm\") pod \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") "
Sep 09 14:37:03.958847 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.958853    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-home\") pod \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") "
Sep 09 14:37:03.959131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.958912    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ml6p\" (UniqueName: \"kubernetes.io/projected/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-kube-api-access-6ml6p\") pod \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") "
Sep 09 14:37:03.959131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.958945    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tls-certs\") pod \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") "
Sep 09 14:37:03.959131 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.958970    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-model-cache\") pod \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\" (UID: \"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63\") "
Sep 09 14:37:03.959285 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.959072    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" (UID: "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:37:03.959285 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.959178    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-home" (OuterVolumeSpecName: "home") pod "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" (UID: "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:37:03.959285 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.959218    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-model-cache" (OuterVolumeSpecName: "model-cache") pod "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" (UID: "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:37:03.959447 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.959380    2532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-model-cache\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:37:03.959447 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.959409    2532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tmp-dir\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:37:03.959447 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.959417    2532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-home\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:37:03.960971 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.960947    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-dshm" (OuterVolumeSpecName: "dshm") pod "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" (UID: "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:37:03.961069 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.960948    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-kube-api-access-6ml6p" (OuterVolumeSpecName: "kube-api-access-6ml6p") pod "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" (UID: "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63"). InnerVolumeSpecName "kube-api-access-6ml6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:37:03.961069 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:03.961001    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" (UID: "50c0bb7e-aaca-4e52-a5ee-42c7c325fd63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:37:04.060598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.060554    2532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-dshm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:37:04.060598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.060594    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ml6p\" (UniqueName: \"kubernetes.io/projected/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-kube-api-access-6ml6p\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:37:04.060598 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.060606    2532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63-tls-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:37:04.679553 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.679516    2532 generic.go:356] "Generic (PLEG): container finished" podID="50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" containerID="3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2" exitCode=0
Sep 09 14:37:04.679956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.679564    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"
Sep 09 14:37:04.679956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.679576    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk" event={"ID":"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63","Type":"ContainerDied","Data":"3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2"}
Sep 09 14:37:04.679956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.679616    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk" event={"ID":"50c0bb7e-aaca-4e52-a5ee-42c7c325fd63","Type":"ContainerDied","Data":"0a156d51974382fc2ff9436985bc06e5c7e9b499a0ecd3038a95438e749b0d82"}
Sep 09 14:37:04.679956 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.679639    2532 scope.go:117] "RemoveContainer" containerID="3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2"
Sep 09 14:37:04.689236 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.689216    2532 scope.go:117] "RemoveContainer" containerID="3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2"
Sep 09 14:37:04.689491 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:37:04.689473    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2\": container with ID starting with 3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2 not found: ID does not exist" containerID="3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2"
Sep 09 14:37:04.689546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.689499    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2"} err="failed to get container status \"3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2\": rpc error: code = NotFound desc = could not find container \"3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2\": container with ID starting with 3ea96126e9d93ae8409f85ab21c977418b78741045b35f4c8a7b3781f545c5c2 not found: ID does not exist"
Sep 09 14:37:04.700595 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.700570    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"]
Sep 09 14:37:04.704061 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.704041    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e565aa/rate-test-model-e565aa-kserve-7c564854c9-hxbpk"]
Sep 09 14:37:04.735077 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:04.735052    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" path="/var/lib/kubelet/pods/50c0bb7e-aaca-4e52-a5ee-42c7c325fd63/volumes"
Sep 09 14:37:07.644293 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:07.644258    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:37:24.024661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.024630    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"]
Sep 09 14:37:24.025177 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.025056    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" containerName="main"
Sep 09 14:37:24.025177 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.025067    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" containerName="main"
Sep 09 14:37:24.025177 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.025171    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="50c0bb7e-aaca-4e52-a5ee-42c7c325fd63" containerName="main"
Sep 09 14:37:24.028551 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.028533    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:37:24.030881 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.030859    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f583d9\""
Sep 09 14:37:24.031032 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.031012    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f583d9-dockercfg-hwwvx\""
Sep 09 14:37:24.037943 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.037915    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"]
Sep 09 14:37:24.134319 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.134290    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59c7\" (UniqueName: \"kubernetes.io/projected/4b94fff6-d4da-48c1-bf1f-3049b83d939f-kube-api-access-q59c7\") pod \"payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb\" (UID: \"4b94fff6-d4da-48c1-bf1f-3049b83d939f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:37:24.134459 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.134341    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b94fff6-d4da-48c1-bf1f-3049b83d939f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb\" (UID: \"4b94fff6-d4da-48c1-bf1f-3049b83d939f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:37:24.235792 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.235755    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59c7\" (UniqueName: \"kubernetes.io/projected/4b94fff6-d4da-48c1-bf1f-3049b83d939f-kube-api-access-q59c7\") pod \"payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb\" (UID: \"4b94fff6-d4da-48c1-bf1f-3049b83d939f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:37:24.235941 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.235824    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b94fff6-d4da-48c1-bf1f-3049b83d939f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb\" (UID: \"4b94fff6-d4da-48c1-bf1f-3049b83d939f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:37:24.236493 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.236475    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b94fff6-d4da-48c1-bf1f-3049b83d939f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb\" (UID: \"4b94fff6-d4da-48c1-bf1f-3049b83d939f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:37:24.244254 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.244235    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59c7\" (UniqueName: \"kubernetes.io/projected/4b94fff6-d4da-48c1-bf1f-3049b83d939f-kube-api-access-q59c7\") pod \"payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb\" (UID: \"4b94fff6-d4da-48c1-bf1f-3049b83d939f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:37:24.340401 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.340333    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:37:24.609872 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.609710    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"]
Sep 09 14:37:24.612537 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:37:24.612507    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b94fff6_d4da_48c1_bf1f_3049b83d939f.slice/crio-050858dc5d4f8550617f742acfdf98235864099d75d4ad91cf3c98e347c2bdb9 WatchSource:0}: Error finding container 050858dc5d4f8550617f742acfdf98235864099d75d4ad91cf3c98e347c2bdb9: Status 404 returned error can't find the container with id 050858dc5d4f8550617f742acfdf98235864099d75d4ad91cf3c98e347c2bdb9
Sep 09 14:37:24.786694 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.786664    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb" event={"ID":"4b94fff6-d4da-48c1-bf1f-3049b83d939f","Type":"ContainerStarted","Data":"a04c4efbcfaa8d97c29a70ad311769fc70517292157769c1f584defa2a8831c0"}
Sep 09 14:37:24.786848 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.786717    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb" event={"ID":"4b94fff6-d4da-48c1-bf1f-3049b83d939f","Type":"ContainerStarted","Data":"050858dc5d4f8550617f742acfdf98235864099d75d4ad91cf3c98e347c2bdb9"}
Sep 09 14:37:24.786848 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.786747    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:37:24.804505 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:24.804463    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb" podStartSLOduration=0.804447387 podStartE2EDuration="804.447387ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:24.802807443 +0000 UTC m=+1578.565602695" watchObservedRunningTime="2026-09-09 14:37:24.804447387 +0000 UTC m=+1578.567242665"
Sep 09 14:37:35.794554 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:35.794525    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:37:52.547740 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:37:52.547700    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb_4b94fff6-d4da-48c1-bf1f-3049b83d939f/payload-processing/0.log"
Sep 09 14:38:38.372853 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:38.372823    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c_1ffbb714-c838-4c10-accd-fdd35ed06a97/payload-processing/0.log"
Sep 09 14:38:38.609476 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:38.609448    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb_4b94fff6-d4da-48c1-bf1f-3049b83d939f/payload-processing/0.log"
Sep 09 14:38:43.905515 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:43.904972    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"]
Sep 09 14:38:43.909395 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:43.909373    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:43.912460 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:43.912428    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-b37ae4-serving-cert\""
Sep 09 14:38:43.920545 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:43.920518    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"]
Sep 09 14:38:44.009864 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.009833    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.010003 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.009885    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a7b8c88-f635-4e03-8bea-c7c788784cf2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.010003 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.009942    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.010003 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.009958    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k74cp\" (UniqueName: \"kubernetes.io/projected/1a7b8c88-f635-4e03-8bea-c7c788784cf2-kube-api-access-k74cp\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.110568 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.110521    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.110568 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.110569    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k74cp\" (UniqueName: \"kubernetes.io/projected/1a7b8c88-f635-4e03-8bea-c7c788784cf2-kube-api-access-k74cp\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.110802 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.110620    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.110802 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.110653    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a7b8c88-f635-4e03-8bea-c7c788784cf2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.111299 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.111275    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a7b8c88-f635-4e03-8bea-c7c788784cf2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.113046 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.113018    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.113165 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.113065    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.118454 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.118430    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k74cp\" (UniqueName: \"kubernetes.io/projected/1a7b8c88-f635-4e03-8bea-c7c788784cf2-kube-api-access-k74cp\") pod \"maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.223759 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.223689    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:44.500511 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.500487    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"]
Sep 09 14:38:44.502200 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:38:44.502174    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7b8c88_f635_4e03_8bea_c7c788784cf2.slice/crio-05f7c9e30c344286c83d65e333f4881fd41986551de0b51063b2e05fff54657e WatchSource:0}: Error finding container 05f7c9e30c344286c83d65e333f4881fd41986551de0b51063b2e05fff54657e: Status 404 returned error can't find the container with id 05f7c9e30c344286c83d65e333f4881fd41986551de0b51063b2e05fff54657e
Sep 09 14:38:44.522445 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.522422    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"]
Sep 09 14:38:44.527388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.527370    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:38:44.529622 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.529601    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-b37ae4-dockercfg-drpfk\""
Sep 09 14:38:44.529903 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.529888    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-b37ae4\""
Sep 09 14:38:44.539009 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.538988    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"]
Sep 09 14:38:44.615092 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.615064    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f8ced0-f445-4e51-b81e-4f20f170a4a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd\" (UID: \"30f8ced0-f445-4e51-b81e-4f20f170a4a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:38:44.615221 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.615134    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2t5c\" (UniqueName: \"kubernetes.io/projected/30f8ced0-f445-4e51-b81e-4f20f170a4a6-kube-api-access-n2t5c\") pod \"payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd\" (UID: \"30f8ced0-f445-4e51-b81e-4f20f170a4a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:38:44.716026 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.716001    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f8ced0-f445-4e51-b81e-4f20f170a4a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd\" (UID: \"30f8ced0-f445-4e51-b81e-4f20f170a4a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:38:44.716159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.716043    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2t5c\" (UniqueName: \"kubernetes.io/projected/30f8ced0-f445-4e51-b81e-4f20f170a4a6-kube-api-access-n2t5c\") pod \"payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd\" (UID: \"30f8ced0-f445-4e51-b81e-4f20f170a4a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:38:44.716655 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.716635    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f8ced0-f445-4e51-b81e-4f20f170a4a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd\" (UID: \"30f8ced0-f445-4e51-b81e-4f20f170a4a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:38:44.724619 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.724596    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2t5c\" (UniqueName: \"kubernetes.io/projected/30f8ced0-f445-4e51-b81e-4f20f170a4a6-kube-api-access-n2t5c\") pod \"payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd\" (UID: \"30f8ced0-f445-4e51-b81e-4f20f170a4a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:38:44.838516 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:44.838433    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:38:45.109000 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:45.108974    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"]
Sep 09 14:38:45.111134 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:38:45.111108    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f8ced0_f445_4e51_b81e_4f20f170a4a6.slice/crio-71f9d8afec13dd8ee2cecacf7598834eb6b30456ac22ecf16f45a85aa651e347 WatchSource:0}: Error finding container 71f9d8afec13dd8ee2cecacf7598834eb6b30456ac22ecf16f45a85aa651e347: Status 404 returned error can't find the container with id 71f9d8afec13dd8ee2cecacf7598834eb6b30456ac22ecf16f45a85aa651e347
Sep 09 14:38:45.196494 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:45.196458    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd" event={"ID":"30f8ced0-f445-4e51-b81e-4f20f170a4a6","Type":"ContainerStarted","Data":"90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7"}
Sep 09 14:38:45.196634 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:45.196522    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd" event={"ID":"30f8ced0-f445-4e51-b81e-4f20f170a4a6","Type":"ContainerStarted","Data":"71f9d8afec13dd8ee2cecacf7598834eb6b30456ac22ecf16f45a85aa651e347"}
Sep 09 14:38:45.196703 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:45.196639    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:38:45.198229 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:45.198206    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc" event={"ID":"1a7b8c88-f635-4e03-8bea-c7c788784cf2","Type":"ContainerStarted","Data":"2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130"}
Sep 09 14:38:45.198342 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:45.198258    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc" event={"ID":"1a7b8c88-f635-4e03-8bea-c7c788784cf2","Type":"ContainerStarted","Data":"05f7c9e30c344286c83d65e333f4881fd41986551de0b51063b2e05fff54657e"}
Sep 09 14:38:45.198342 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:45.198280    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:45.214824 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:45.214717    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd" podStartSLOduration=1.214698361 podStartE2EDuration="1.214698361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:45.2139986 +0000 UTC m=+1658.976793840" watchObservedRunningTime="2026-09-09 14:38:45.214698361 +0000 UTC m=+1658.977493599"
Sep 09 14:38:45.231712 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:45.231536    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc" podStartSLOduration=1.767559616 podStartE2EDuration="2.231517224s" totalImagesPullingTime="463.957608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:45.230977307 +0000 UTC m=+1658.993772543" watchObservedRunningTime="2026-09-09 14:38:45.231517224 +0000 UTC m=+1658.994312466"
Sep 09 14:38:51.217195 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:51.217162    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:38:56.213695 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:38:56.213666    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:39:14.301916 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.301880    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"]
Sep 09 14:39:14.302563 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.302150    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc" podUID="1a7b8c88-f635-4e03-8bea-c7c788784cf2" containerName="maas-api" containerID="cri-o://2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130" gracePeriod=30
Sep 09 14:39:14.355899 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.355872    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"]
Sep 09 14:39:14.356166 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.356141    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd" podUID="30f8ced0-f445-4e51-b81e-4f20f170a4a6" containerName="payload-processing" containerID="cri-o://90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7" gracePeriod=30
Sep 09 14:39:14.602079 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.602056    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:39:14.631696 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.631674    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:39:14.675163 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.675138    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k74cp\" (UniqueName: \"kubernetes.io/projected/1a7b8c88-f635-4e03-8bea-c7c788784cf2-kube-api-access-k74cp\") pod \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") "
Sep 09 14:39:14.675670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.675228    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-maas-api-tls\") pod \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") "
Sep 09 14:39:14.675670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.675275    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a7b8c88-f635-4e03-8bea-c7c788784cf2-openshift-service-ca\") pod \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") "
Sep 09 14:39:14.675670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.675310    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-metrics-certs\") pod \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\" (UID: \"1a7b8c88-f635-4e03-8bea-c7c788784cf2\") "
Sep 09 14:39:14.675670 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.675596    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7b8c88-f635-4e03-8bea-c7c788784cf2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a7b8c88-f635-4e03-8bea-c7c788784cf2" (UID: "1a7b8c88-f635-4e03-8bea-c7c788784cf2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:39:14.677387 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.677360    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a7b8c88-f635-4e03-8bea-c7c788784cf2" (UID: "1a7b8c88-f635-4e03-8bea-c7c788784cf2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:39:14.677468 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.677374    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1a7b8c88-f635-4e03-8bea-c7c788784cf2" (UID: "1a7b8c88-f635-4e03-8bea-c7c788784cf2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:39:14.677582 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.677563    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7b8c88-f635-4e03-8bea-c7c788784cf2-kube-api-access-k74cp" (OuterVolumeSpecName: "kube-api-access-k74cp") pod "1a7b8c88-f635-4e03-8bea-c7c788784cf2" (UID: "1a7b8c88-f635-4e03-8bea-c7c788784cf2"). InnerVolumeSpecName "kube-api-access-k74cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:39:14.776700 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.776674    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f8ced0-f445-4e51-b81e-4f20f170a4a6-plugins-config-volume\") pod \"30f8ced0-f445-4e51-b81e-4f20f170a4a6\" (UID: \"30f8ced0-f445-4e51-b81e-4f20f170a4a6\") "
Sep 09 14:39:14.776821 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.776769    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2t5c\" (UniqueName: \"kubernetes.io/projected/30f8ced0-f445-4e51-b81e-4f20f170a4a6-kube-api-access-n2t5c\") pod \"30f8ced0-f445-4e51-b81e-4f20f170a4a6\" (UID: \"30f8ced0-f445-4e51-b81e-4f20f170a4a6\") "
Sep 09 14:39:14.777062 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.777031    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f8ced0-f445-4e51-b81e-4f20f170a4a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30f8ced0-f445-4e51-b81e-4f20f170a4a6" (UID: "30f8ced0-f445-4e51-b81e-4f20f170a4a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:39:14.777178 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.777069    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:14.777178 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.777089    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a7b8c88-f635-4e03-8bea-c7c788784cf2-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:14.777178 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.777124    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a7b8c88-f635-4e03-8bea-c7c788784cf2-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:14.777178 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.777134    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k74cp\" (UniqueName: \"kubernetes.io/projected/1a7b8c88-f635-4e03-8bea-c7c788784cf2-kube-api-access-k74cp\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:14.778586 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.778569    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f8ced0-f445-4e51-b81e-4f20f170a4a6-kube-api-access-n2t5c" (OuterVolumeSpecName: "kube-api-access-n2t5c") pod "30f8ced0-f445-4e51-b81e-4f20f170a4a6" (UID: "30f8ced0-f445-4e51-b81e-4f20f170a4a6"). InnerVolumeSpecName "kube-api-access-n2t5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:39:14.878281 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.878231    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f8ced0-f445-4e51-b81e-4f20f170a4a6-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:14.878281 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:14.878252    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2t5c\" (UniqueName: \"kubernetes.io/projected/30f8ced0-f445-4e51-b81e-4f20f170a4a6-kube-api-access-n2t5c\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:15.367683 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.367651    2532 generic.go:356] "Generic (PLEG): container finished" podID="30f8ced0-f445-4e51-b81e-4f20f170a4a6" containerID="90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7" exitCode=0
Sep 09 14:39:15.368122 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.367691    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"
Sep 09 14:39:15.368122 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.367734    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd" event={"ID":"30f8ced0-f445-4e51-b81e-4f20f170a4a6","Type":"ContainerDied","Data":"90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7"}
Sep 09 14:39:15.368122 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.367788    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd" event={"ID":"30f8ced0-f445-4e51-b81e-4f20f170a4a6","Type":"ContainerDied","Data":"71f9d8afec13dd8ee2cecacf7598834eb6b30456ac22ecf16f45a85aa651e347"}
Sep 09 14:39:15.368122 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.367817    2532 scope.go:117] "RemoveContainer" containerID="90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7"
Sep 09 14:39:15.369337 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.369316    2532 generic.go:356] "Generic (PLEG): container finished" podID="1a7b8c88-f635-4e03-8bea-c7c788784cf2" containerID="2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130" exitCode=0
Sep 09 14:39:15.369433 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.369410    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"
Sep 09 14:39:15.369433 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.369428    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc" event={"ID":"1a7b8c88-f635-4e03-8bea-c7c788784cf2","Type":"ContainerDied","Data":"2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130"}
Sep 09 14:39:15.369610 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.369468    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc" event={"ID":"1a7b8c88-f635-4e03-8bea-c7c788784cf2","Type":"ContainerDied","Data":"05f7c9e30c344286c83d65e333f4881fd41986551de0b51063b2e05fff54657e"}
Sep 09 14:39:15.377611 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.377593    2532 scope.go:117] "RemoveContainer" containerID="90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7"
Sep 09 14:39:15.377924 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:39:15.377903    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7\": container with ID starting with 90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7 not found: ID does not exist" containerID="90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7"
Sep 09 14:39:15.377981 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.377933    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7"} err="failed to get container status \"90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7\": rpc error: code = NotFound desc = could not find container \"90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7\": container with ID starting with 90c3b5f3a254dd0f6a486f269f86870717bfd2e6dee4676fd267b575a0a8b9d7 not found: ID does not exist"
Sep 09 14:39:15.377981 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.377949    2532 scope.go:117] "RemoveContainer" containerID="2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130"
Sep 09 14:39:15.386225 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.386147    2532 scope.go:117] "RemoveContainer" containerID="2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130"
Sep 09 14:39:15.386528 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:39:15.386493    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130\": container with ID starting with 2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130 not found: ID does not exist" containerID="2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130"
Sep 09 14:39:15.386528 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.386523    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130"} err="failed to get container status \"2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130\": rpc error: code = NotFound desc = could not find container \"2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130\": container with ID starting with 2513d2d4f2f8368deac051f9de73f17ed67680611c0ecff75b70d543b8c97130 not found: ID does not exist"
Sep 09 14:39:15.388575 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.388558    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"]
Sep 09 14:39:15.391926 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.391906    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b37ae4-db88dc9b6-6fsqc"]
Sep 09 14:39:15.407129 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.407082    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"]
Sep 09 14:39:15.417073 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:15.412561    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b37ae4-75548d579-6twpd"]
Sep 09 14:39:16.734272 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:16.734240    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7b8c88-f635-4e03-8bea-c7c788784cf2" path="/var/lib/kubelet/pods/1a7b8c88-f635-4e03-8bea-c7c788784cf2/volumes"
Sep 09 14:39:16.734634 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:16.734577    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30f8ced0-f445-4e51-b81e-4f20f170a4a6" path="/var/lib/kubelet/pods/30f8ced0-f445-4e51-b81e-4f20f170a4a6/volumes"
Sep 09 14:39:40.791133 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:40.791010    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"]
Sep 09 14:39:40.795985 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:39:40.795958    2532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-serving-cert: secret "maas-api-e2e-ipp-a-ea13e2-serving-cert" not found
Sep 09 14:39:40.796145 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:39:40.796042    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-maas-api-tls podName:a483f9a3-6b69-4838-8e4c-96916f1a5546 nodeName:}" failed. No retries permitted until 2026-09-09 14:39:41.296020235 +0000 UTC m=+1715.058815467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-maas-api-tls") pod "maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592" (UID: "a483f9a3-6b69-4838-8e4c-96916f1a5546") : secret "maas-api-e2e-ipp-a-ea13e2-serving-cert" not found
Sep 09 14:39:40.841971 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:40.841937    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"]
Sep 09 14:39:41.300167 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:39:41.300128    2532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-serving-cert: secret "maas-api-e2e-ipp-a-ea13e2-serving-cert" not found
Sep 09 14:39:41.300376 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:39:41.300222    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-maas-api-tls podName:a483f9a3-6b69-4838-8e4c-96916f1a5546 nodeName:}" failed. No retries permitted until 2026-09-09 14:39:42.300201402 +0000 UTC m=+1716.062996621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-maas-api-tls") pod "maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592" (UID: "a483f9a3-6b69-4838-8e4c-96916f1a5546") : secret "maas-api-e2e-ipp-a-ea13e2-serving-cert" not found
Sep 09 14:39:41.504873 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.504818    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c" podUID="1ffbb714-c838-4c10-accd-fdd35ed06a97" containerName="payload-processing" containerID="cri-o://3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6" gracePeriod=30
Sep 09 14:39:41.504873 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.504852    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592" podUID="a483f9a3-6b69-4838-8e4c-96916f1a5546" containerName="maas-api" containerID="cri-o://4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927" gracePeriod=30
Sep 09 14:39:41.751742 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.751720    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:39:41.781548 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.781522    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:39:41.803180 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.803155    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffbb714-c838-4c10-accd-fdd35ed06a97-plugins-config-volume\") pod \"1ffbb714-c838-4c10-accd-fdd35ed06a97\" (UID: \"1ffbb714-c838-4c10-accd-fdd35ed06a97\") "
Sep 09 14:39:41.803572 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.803216    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-metrics-certs\") pod \"a483f9a3-6b69-4838-8e4c-96916f1a5546\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") "
Sep 09 14:39:41.803572 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.803316    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lcg2\" (UniqueName: \"kubernetes.io/projected/a483f9a3-6b69-4838-8e4c-96916f1a5546-kube-api-access-7lcg2\") pod \"a483f9a3-6b69-4838-8e4c-96916f1a5546\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") "
Sep 09 14:39:41.803572 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.803355    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a483f9a3-6b69-4838-8e4c-96916f1a5546-openshift-service-ca\") pod \"a483f9a3-6b69-4838-8e4c-96916f1a5546\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") "
Sep 09 14:39:41.803572 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.803391    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-maas-api-tls\") pod \"a483f9a3-6b69-4838-8e4c-96916f1a5546\" (UID: \"a483f9a3-6b69-4838-8e4c-96916f1a5546\") "
Sep 09 14:39:41.803572 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.803426    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4bzg\" (UniqueName: \"kubernetes.io/projected/1ffbb714-c838-4c10-accd-fdd35ed06a97-kube-api-access-j4bzg\") pod \"1ffbb714-c838-4c10-accd-fdd35ed06a97\" (UID: \"1ffbb714-c838-4c10-accd-fdd35ed06a97\") "
Sep 09 14:39:41.803572 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.803532    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffbb714-c838-4c10-accd-fdd35ed06a97-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ffbb714-c838-4c10-accd-fdd35ed06a97" (UID: "1ffbb714-c838-4c10-accd-fdd35ed06a97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:39:41.803919 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.803746    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a483f9a3-6b69-4838-8e4c-96916f1a5546-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a483f9a3-6b69-4838-8e4c-96916f1a5546" (UID: "a483f9a3-6b69-4838-8e4c-96916f1a5546"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:39:41.803983 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.803927    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a483f9a3-6b69-4838-8e4c-96916f1a5546-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:41.803983 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.803950    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffbb714-c838-4c10-accd-fdd35ed06a97-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:41.806123 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.806070    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a483f9a3-6b69-4838-8e4c-96916f1a5546-kube-api-access-7lcg2" (OuterVolumeSpecName: "kube-api-access-7lcg2") pod "a483f9a3-6b69-4838-8e4c-96916f1a5546" (UID: "a483f9a3-6b69-4838-8e4c-96916f1a5546"). InnerVolumeSpecName "kube-api-access-7lcg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:39:41.806231 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.806153    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a483f9a3-6b69-4838-8e4c-96916f1a5546" (UID: "a483f9a3-6b69-4838-8e4c-96916f1a5546"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:39:41.806329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.806303    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a483f9a3-6b69-4838-8e4c-96916f1a5546" (UID: "a483f9a3-6b69-4838-8e4c-96916f1a5546"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:39:41.806441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.806380    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffbb714-c838-4c10-accd-fdd35ed06a97-kube-api-access-j4bzg" (OuterVolumeSpecName: "kube-api-access-j4bzg") pod "1ffbb714-c838-4c10-accd-fdd35ed06a97" (UID: "1ffbb714-c838-4c10-accd-fdd35ed06a97"). InnerVolumeSpecName "kube-api-access-j4bzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:39:41.905422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.905359    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:41.905422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.905381    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lcg2\" (UniqueName: \"kubernetes.io/projected/a483f9a3-6b69-4838-8e4c-96916f1a5546-kube-api-access-7lcg2\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:41.905422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.905391    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a483f9a3-6b69-4838-8e4c-96916f1a5546-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:41.905422 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:41.905399    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4bzg\" (UniqueName: \"kubernetes.io/projected/1ffbb714-c838-4c10-accd-fdd35ed06a97-kube-api-access-j4bzg\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:39:42.510952 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.510916    2532 generic.go:356] "Generic (PLEG): container finished" podID="a483f9a3-6b69-4838-8e4c-96916f1a5546" containerID="4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927" exitCode=0
Sep 09 14:39:42.511142 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.510958    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"
Sep 09 14:39:42.511142 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.510997    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592" event={"ID":"a483f9a3-6b69-4838-8e4c-96916f1a5546","Type":"ContainerDied","Data":"4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927"}
Sep 09 14:39:42.511142 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.511058    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592" event={"ID":"a483f9a3-6b69-4838-8e4c-96916f1a5546","Type":"ContainerDied","Data":"0976461d7ec2c3c456465ff4cfeaf1df2e947fc0ce983bc43f0858923758c42c"}
Sep 09 14:39:42.511142 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.511092    2532 scope.go:117] "RemoveContainer" containerID="4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927"
Sep 09 14:39:42.512506 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.512485    2532 generic.go:356] "Generic (PLEG): container finished" podID="1ffbb714-c838-4c10-accd-fdd35ed06a97" containerID="3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6" exitCode=0
Sep 09 14:39:42.512578 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.512527    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"
Sep 09 14:39:42.512628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.512570    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c" event={"ID":"1ffbb714-c838-4c10-accd-fdd35ed06a97","Type":"ContainerDied","Data":"3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6"}
Sep 09 14:39:42.512628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.512620    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c" event={"ID":"1ffbb714-c838-4c10-accd-fdd35ed06a97","Type":"ContainerDied","Data":"7a858a7b8565268b0ea1055a8c47887c132137618ca654f43d01dd30d6ed3451"}
Sep 09 14:39:42.531444 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.531426    2532 scope.go:117] "RemoveContainer" containerID="4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927"
Sep 09 14:39:42.531707 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:39:42.531690    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927\": container with ID starting with 4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927 not found: ID does not exist" containerID="4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927"
Sep 09 14:39:42.531789 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.531718    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927"} err="failed to get container status \"4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927\": rpc error: code = NotFound desc = could not find container \"4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927\": container with ID starting with 4114a0ab9694c89168b1140cb2229f5ffc7250e112e480fc94ffe8086e417927 not found: ID does not exist"
Sep 09 14:39:42.531789 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.531740    2532 scope.go:117] "RemoveContainer" containerID="3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6"
Sep 09 14:39:42.539878 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.539862    2532 scope.go:117] "RemoveContainer" containerID="3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6"
Sep 09 14:39:42.540146 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:39:42.540126    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6\": container with ID starting with 3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6 not found: ID does not exist" containerID="3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6"
Sep 09 14:39:42.540238 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.540149    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6"} err="failed to get container status \"3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6\": rpc error: code = NotFound desc = could not find container \"3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6\": container with ID starting with 3b28fb930f0f6212112819f2e1ed3640f5ae6da1532d5d7363c111a95fc179c6 not found: ID does not exist"
Sep 09 14:39:42.542720 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.542701    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"]
Sep 09 14:39:42.546472 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.546452    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea13e2-559cdbb688-s6592"]
Sep 09 14:39:42.555941 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.555919    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"]
Sep 09 14:39:42.559854 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.559825    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea13e2-86f768db75-2ws7c"]
Sep 09 14:39:42.734804 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.734777    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ffbb714-c838-4c10-accd-fdd35ed06a97" path="/var/lib/kubelet/pods/1ffbb714-c838-4c10-accd-fdd35ed06a97/volumes"
Sep 09 14:39:42.735189 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:39:42.735174    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a483f9a3-6b69-4838-8e4c-96916f1a5546" path="/var/lib/kubelet/pods/a483f9a3-6b69-4838-8e4c-96916f1a5546/volumes"
Sep 09 14:40:46.680571 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:46.680539    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"]
Sep 09 14:40:46.681046 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:46.680735    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb" podUID="4b94fff6-d4da-48c1-bf1f-3049b83d939f" containerName="payload-processing" containerID="cri-o://a04c4efbcfaa8d97c29a70ad311769fc70517292157769c1f584defa2a8831c0" gracePeriod=30
Sep 09 14:40:46.843414 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:46.843383    2532 generic.go:356] "Generic (PLEG): container finished" podID="4b94fff6-d4da-48c1-bf1f-3049b83d939f" containerID="a04c4efbcfaa8d97c29a70ad311769fc70517292157769c1f584defa2a8831c0" exitCode=0
Sep 09 14:40:46.843562 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:46.843460    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb" event={"ID":"4b94fff6-d4da-48c1-bf1f-3049b83d939f","Type":"ContainerDied","Data":"a04c4efbcfaa8d97c29a70ad311769fc70517292157769c1f584defa2a8831c0"}
Sep 09 14:40:46.937893 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:46.937834    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:40:47.054141 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.054110    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q59c7\" (UniqueName: \"kubernetes.io/projected/4b94fff6-d4da-48c1-bf1f-3049b83d939f-kube-api-access-q59c7\") pod \"4b94fff6-d4da-48c1-bf1f-3049b83d939f\" (UID: \"4b94fff6-d4da-48c1-bf1f-3049b83d939f\") "
Sep 09 14:40:47.054284 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.054151    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b94fff6-d4da-48c1-bf1f-3049b83d939f-plugins-config-volume\") pod \"4b94fff6-d4da-48c1-bf1f-3049b83d939f\" (UID: \"4b94fff6-d4da-48c1-bf1f-3049b83d939f\") "
Sep 09 14:40:47.054441 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.054416    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b94fff6-d4da-48c1-bf1f-3049b83d939f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b94fff6-d4da-48c1-bf1f-3049b83d939f" (UID: "4b94fff6-d4da-48c1-bf1f-3049b83d939f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:40:47.054511 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.054499    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b94fff6-d4da-48c1-bf1f-3049b83d939f-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:40:47.056007 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.055987    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b94fff6-d4da-48c1-bf1f-3049b83d939f-kube-api-access-q59c7" (OuterVolumeSpecName: "kube-api-access-q59c7") pod "4b94fff6-d4da-48c1-bf1f-3049b83d939f" (UID: "4b94fff6-d4da-48c1-bf1f-3049b83d939f"). InnerVolumeSpecName "kube-api-access-q59c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:40:47.155137 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.155086    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q59c7\" (UniqueName: \"kubernetes.io/projected/4b94fff6-d4da-48c1-bf1f-3049b83d939f-kube-api-access-q59c7\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:40:47.849432 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.849402    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"
Sep 09 14:40:47.849875 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.849433    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb" event={"ID":"4b94fff6-d4da-48c1-bf1f-3049b83d939f","Type":"ContainerDied","Data":"050858dc5d4f8550617f742acfdf98235864099d75d4ad91cf3c98e347c2bdb9"}
Sep 09 14:40:47.849875 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.849499    2532 scope.go:117] "RemoveContainer" containerID="a04c4efbcfaa8d97c29a70ad311769fc70517292157769c1f584defa2a8831c0"
Sep 09 14:40:47.873572 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.873549    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"]
Sep 09 14:40:47.877754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:47.877732    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f583d9-5f4489f4d5-hklfb"]
Sep 09 14:40:48.734358 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:40:48.734324    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b94fff6-d4da-48c1-bf1f-3049b83d939f" path="/var/lib/kubelet/pods/4b94fff6-d4da-48c1-bf1f-3049b83d939f/volumes"
Sep 09 14:41:04.918969 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:04.918935    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"]
Sep 09 14:41:04.919375 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:04.919179    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26" podUID="c4e72623-88f1-4eff-8058-8c60f72e295e" containerName="payload-pre-processing" containerID="cri-o://36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050" gracePeriod=30
Sep 09 14:41:05.176769 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.176711    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:41:05.312296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.312267    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq8n6\" (UniqueName: \"kubernetes.io/projected/c4e72623-88f1-4eff-8058-8c60f72e295e-kube-api-access-gq8n6\") pod \"c4e72623-88f1-4eff-8058-8c60f72e295e\" (UID: \"c4e72623-88f1-4eff-8058-8c60f72e295e\") "
Sep 09 14:41:05.312434 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.312361    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e72623-88f1-4eff-8058-8c60f72e295e-plugins-config-volume\") pod \"c4e72623-88f1-4eff-8058-8c60f72e295e\" (UID: \"c4e72623-88f1-4eff-8058-8c60f72e295e\") "
Sep 09 14:41:05.312723 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.312694    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e72623-88f1-4eff-8058-8c60f72e295e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4e72623-88f1-4eff-8058-8c60f72e295e" (UID: "c4e72623-88f1-4eff-8058-8c60f72e295e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:41:05.314260 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.314238    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e72623-88f1-4eff-8058-8c60f72e295e-kube-api-access-gq8n6" (OuterVolumeSpecName: "kube-api-access-gq8n6") pod "c4e72623-88f1-4eff-8058-8c60f72e295e" (UID: "c4e72623-88f1-4eff-8058-8c60f72e295e"). InnerVolumeSpecName "kube-api-access-gq8n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:41:05.413289 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.413265    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq8n6\" (UniqueName: \"kubernetes.io/projected/c4e72623-88f1-4eff-8058-8c60f72e295e-kube-api-access-gq8n6\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:41:05.413289 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.413286    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e72623-88f1-4eff-8058-8c60f72e295e-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:41:05.944712 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.944677    2532 generic.go:356] "Generic (PLEG): container finished" podID="c4e72623-88f1-4eff-8058-8c60f72e295e" containerID="36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050" exitCode=0
Sep 09 14:41:05.945061 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.944722    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"
Sep 09 14:41:05.945061 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.944758    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26" event={"ID":"c4e72623-88f1-4eff-8058-8c60f72e295e","Type":"ContainerDied","Data":"36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050"}
Sep 09 14:41:05.945061 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.944808    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26" event={"ID":"c4e72623-88f1-4eff-8058-8c60f72e295e","Type":"ContainerDied","Data":"851e808a313ae6a49cbcd8050b38dddaa53dbb6b036b6eba30dc1810df84abec"}
Sep 09 14:41:05.945061 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.944831    2532 scope.go:117] "RemoveContainer" containerID="36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050"
Sep 09 14:41:05.956151 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.956132    2532 scope.go:117] "RemoveContainer" containerID="36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050"
Sep 09 14:41:05.956434 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:41:05.956388    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050\": container with ID starting with 36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050 not found: ID does not exist" containerID="36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050"
Sep 09 14:41:05.956527 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.956441    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050"} err="failed to get container status \"36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050\": rpc error: code = NotFound desc = could not find container \"36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050\": container with ID starting with 36855fa6a9edcc784b10457d19feabff921515bed1e15ad9534207e471e87050 not found: ID does not exist"
Sep 09 14:41:05.967424 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.967400    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"]
Sep 09 14:41:05.971202 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:05.971181    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e565aa-68f6b6449d-jnz26"]
Sep 09 14:41:06.734790 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:06.734755    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e72623-88f1-4eff-8058-8c60f72e295e" path="/var/lib/kubelet/pods/c4e72623-88f1-4eff-8058-8c60f72e295e/volumes"
Sep 09 14:41:06.808273 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:06.808251    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:41:06.811845 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:06.811822    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:41:06.987688 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:06.987619    2532 scope.go:117] "RemoveContainer" containerID="326e238eccb9c029eca09d2c33dd7fbf2aec38458c539307af7f410b0a0893e1"
Sep 09 14:41:23.019754 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.019716    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"]
Sep 09 14:41:23.020384 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.019993    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl" podUID="c539c285-1bb3-4ed3-9703-a1e2862d499d" containerName="maas-api" containerID="cri-o://f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2" gracePeriod=30
Sep 09 14:41:23.079689 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.079656    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"]
Sep 09 14:41:23.079946 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.079889    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8" podUID="7a24e00d-c292-434a-86bc-ad1c3a758a17" containerName="payload-pre-processing" containerID="cri-o://19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff" gracePeriod=30
Sep 09 14:41:23.315866 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.315843    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:41:23.347508 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.347484    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:41:23.475093 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.475060    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c539c285-1bb3-4ed3-9703-a1e2862d499d-openshift-service-ca\") pod \"c539c285-1bb3-4ed3-9703-a1e2862d499d\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") "
Sep 09 14:41:23.475301 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.475130    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-maas-api-tls\") pod \"c539c285-1bb3-4ed3-9703-a1e2862d499d\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") "
Sep 09 14:41:23.475301 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.475172    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-metrics-certs\") pod \"c539c285-1bb3-4ed3-9703-a1e2862d499d\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") "
Sep 09 14:41:23.475301 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.475231    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vrm5\" (UniqueName: \"kubernetes.io/projected/7a24e00d-c292-434a-86bc-ad1c3a758a17-kube-api-access-4vrm5\") pod \"7a24e00d-c292-434a-86bc-ad1c3a758a17\" (UID: \"7a24e00d-c292-434a-86bc-ad1c3a758a17\") "
Sep 09 14:41:23.475301 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.475266    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a24e00d-c292-434a-86bc-ad1c3a758a17-plugins-config-volume\") pod \"7a24e00d-c292-434a-86bc-ad1c3a758a17\" (UID: \"7a24e00d-c292-434a-86bc-ad1c3a758a17\") "
Sep 09 14:41:23.475301 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.475292    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmcv7\" (UniqueName: \"kubernetes.io/projected/c539c285-1bb3-4ed3-9703-a1e2862d499d-kube-api-access-bmcv7\") pod \"c539c285-1bb3-4ed3-9703-a1e2862d499d\" (UID: \"c539c285-1bb3-4ed3-9703-a1e2862d499d\") "
Sep 09 14:41:23.475566 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.475440    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c539c285-1bb3-4ed3-9703-a1e2862d499d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c539c285-1bb3-4ed3-9703-a1e2862d499d" (UID: "c539c285-1bb3-4ed3-9703-a1e2862d499d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:41:23.475677 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.475655    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c539c285-1bb3-4ed3-9703-a1e2862d499d-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:41:23.475677 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.475656    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a24e00d-c292-434a-86bc-ad1c3a758a17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a24e00d-c292-434a-86bc-ad1c3a758a17" (UID: "7a24e00d-c292-434a-86bc-ad1c3a758a17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:41:23.477400 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.477377    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c539c285-1bb3-4ed3-9703-a1e2862d499d" (UID: "c539c285-1bb3-4ed3-9703-a1e2862d499d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:41:23.477502 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.477397    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c539c285-1bb3-4ed3-9703-a1e2862d499d" (UID: "c539c285-1bb3-4ed3-9703-a1e2862d499d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:41:23.477502 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.477439    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a24e00d-c292-434a-86bc-ad1c3a758a17-kube-api-access-4vrm5" (OuterVolumeSpecName: "kube-api-access-4vrm5") pod "7a24e00d-c292-434a-86bc-ad1c3a758a17" (UID: "7a24e00d-c292-434a-86bc-ad1c3a758a17"). InnerVolumeSpecName "kube-api-access-4vrm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:41:23.477502 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.477465    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c539c285-1bb3-4ed3-9703-a1e2862d499d-kube-api-access-bmcv7" (OuterVolumeSpecName: "kube-api-access-bmcv7") pod "c539c285-1bb3-4ed3-9703-a1e2862d499d" (UID: "c539c285-1bb3-4ed3-9703-a1e2862d499d"). InnerVolumeSpecName "kube-api-access-bmcv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:41:23.576149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.576075    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vrm5\" (UniqueName: \"kubernetes.io/projected/7a24e00d-c292-434a-86bc-ad1c3a758a17-kube-api-access-4vrm5\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:41:23.576149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.576115    2532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a24e00d-c292-434a-86bc-ad1c3a758a17-plugins-config-volume\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:41:23.576149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.576126    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmcv7\" (UniqueName: \"kubernetes.io/projected/c539c285-1bb3-4ed3-9703-a1e2862d499d-kube-api-access-bmcv7\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:41:23.576149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.576135    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:41:23.576149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:23.576143    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c539c285-1bb3-4ed3-9703-a1e2862d499d-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:41:24.033705 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.033673    2532 generic.go:356] "Generic (PLEG): container finished" podID="7a24e00d-c292-434a-86bc-ad1c3a758a17" containerID="19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff" exitCode=0
Sep 09 14:41:24.034170 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.033713    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"
Sep 09 14:41:24.034170 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.033759    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8" event={"ID":"7a24e00d-c292-434a-86bc-ad1c3a758a17","Type":"ContainerDied","Data":"19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff"}
Sep 09 14:41:24.034170 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.033813    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8" event={"ID":"7a24e00d-c292-434a-86bc-ad1c3a758a17","Type":"ContainerDied","Data":"9d33355fb5db8271e7ae94ef7b1d18b974990d8576d19340511e97b732b1d8a8"}
Sep 09 14:41:24.034170 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.033845    2532 scope.go:117] "RemoveContainer" containerID="19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff"
Sep 09 14:41:24.035296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.035276    2532 generic.go:356] "Generic (PLEG): container finished" podID="c539c285-1bb3-4ed3-9703-a1e2862d499d" containerID="f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2" exitCode=0
Sep 09 14:41:24.035405 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.035335    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"
Sep 09 14:41:24.035405 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.035344    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl" event={"ID":"c539c285-1bb3-4ed3-9703-a1e2862d499d","Type":"ContainerDied","Data":"f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2"}
Sep 09 14:41:24.035507 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.035416    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl" event={"ID":"c539c285-1bb3-4ed3-9703-a1e2862d499d","Type":"ContainerDied","Data":"9caa15ef44f2e3d4286cf4aa1a9c50940ce352e1086e45a1cb0d94b0ebbccab3"}
Sep 09 14:41:24.046331 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.046303    2532 scope.go:117] "RemoveContainer" containerID="19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff"
Sep 09 14:41:24.046601 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:41:24.046583    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff\": container with ID starting with 19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff not found: ID does not exist" containerID="19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff"
Sep 09 14:41:24.046661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.046609    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff"} err="failed to get container status \"19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff\": rpc error: code = NotFound desc = could not find container \"19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff\": container with ID starting with 19ebdc4aeb46c2f83534e01de79c69ed4f27676a63e1ee0f9186353cded9a2ff not found: ID does not exist"
Sep 09 14:41:24.046661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.046626    2532 scope.go:117] "RemoveContainer" containerID="f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2"
Sep 09 14:41:24.053951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.053934    2532 scope.go:117] "RemoveContainer" containerID="f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2"
Sep 09 14:41:24.054199 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:41:24.054181    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2\": container with ID starting with f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2 not found: ID does not exist" containerID="f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2"
Sep 09 14:41:24.054271 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.054206    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2"} err="failed to get container status \"f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2\": rpc error: code = NotFound desc = could not find container \"f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2\": container with ID starting with f8d14c7399d9301d11e7dd8735047281d4cc93db0cd9404bc3e6f858538c2dc2 not found: ID does not exist"
Sep 09 14:41:24.057809 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.057786    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"]
Sep 09 14:41:24.069837 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.069817    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c5e2d3-6d598594-4lxb8"]
Sep 09 14:41:24.084207 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.084182    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"]
Sep 09 14:41:24.085831 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.085814    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c5e2d3-74454c8ccf-nt4zl"]
Sep 09 14:41:24.733845 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.733814    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a24e00d-c292-434a-86bc-ad1c3a758a17" path="/var/lib/kubelet/pods/7a24e00d-c292-434a-86bc-ad1c3a758a17/volumes"
Sep 09 14:41:24.734223 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:41:24.734207    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c539c285-1bb3-4ed3-9703-a1e2862d499d" path="/var/lib/kubelet/pods/c539c285-1bb3-4ed3-9703-a1e2862d499d/volumes"
Sep 09 14:45:52.690927 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.690847    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"]
Sep 09 14:45:52.691489 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691430    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7b8c88-f635-4e03-8bea-c7c788784cf2" containerName="maas-api"
Sep 09 14:45:52.691489 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691450    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7b8c88-f635-4e03-8bea-c7c788784cf2" containerName="maas-api"
Sep 09 14:45:52.691489 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691470    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e72623-88f1-4eff-8058-8c60f72e295e" containerName="payload-pre-processing"
Sep 09 14:45:52.691489 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691479    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e72623-88f1-4eff-8058-8c60f72e295e" containerName="payload-pre-processing"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691493    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c539c285-1bb3-4ed3-9703-a1e2862d499d" containerName="maas-api"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691501    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="c539c285-1bb3-4ed3-9703-a1e2862d499d" containerName="maas-api"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691520    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ffbb714-c838-4c10-accd-fdd35ed06a97" containerName="payload-processing"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691528    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ffbb714-c838-4c10-accd-fdd35ed06a97" containerName="payload-processing"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691539    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a483f9a3-6b69-4838-8e4c-96916f1a5546" containerName="maas-api"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691547    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="a483f9a3-6b69-4838-8e4c-96916f1a5546" containerName="maas-api"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691565    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a24e00d-c292-434a-86bc-ad1c3a758a17" containerName="payload-pre-processing"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691573    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a24e00d-c292-434a-86bc-ad1c3a758a17" containerName="payload-pre-processing"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691583    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f8ced0-f445-4e51-b81e-4f20f170a4a6" containerName="payload-processing"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691592    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f8ced0-f445-4e51-b81e-4f20f170a4a6" containerName="payload-processing"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691612    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b94fff6-d4da-48c1-bf1f-3049b83d939f" containerName="payload-processing"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691620    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b94fff6-d4da-48c1-bf1f-3049b83d939f" containerName="payload-processing"
Sep 09 14:45:52.691719 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691711    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b94fff6-d4da-48c1-bf1f-3049b83d939f" containerName="payload-processing"
Sep 09 14:45:52.692388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691727    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ffbb714-c838-4c10-accd-fdd35ed06a97" containerName="payload-processing"
Sep 09 14:45:52.692388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691737    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="30f8ced0-f445-4e51-b81e-4f20f170a4a6" containerName="payload-processing"
Sep 09 14:45:52.692388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691751    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e72623-88f1-4eff-8058-8c60f72e295e" containerName="payload-pre-processing"
Sep 09 14:45:52.692388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691762    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a24e00d-c292-434a-86bc-ad1c3a758a17" containerName="payload-pre-processing"
Sep 09 14:45:52.692388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691770    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7b8c88-f635-4e03-8bea-c7c788784cf2" containerName="maas-api"
Sep 09 14:45:52.692388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691789    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="c539c285-1bb3-4ed3-9703-a1e2862d499d" containerName="maas-api"
Sep 09 14:45:52.692388 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.691798    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="a483f9a3-6b69-4838-8e4c-96916f1a5546" containerName="maas-api"
Sep 09 14:45:52.695268 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.695241    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.697571 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.697550    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e34b3797-kserve-self-signed-certs\""
Sep 09 14:45:52.704746 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.704708    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"]
Sep 09 14:45:52.750023 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.749994    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-dshm\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.750165 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.750032    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c02de27-f10b-4aff-80de-228105a458c2-tls-certs\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.750165 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.750137    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-home\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.750267 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.750170    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-model-cache\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.750267 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.750208    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-tmp-dir\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.750337 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.750270    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhxjn\" (UniqueName: \"kubernetes.io/projected/6c02de27-f10b-4aff-80de-228105a458c2-kube-api-access-lhxjn\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.851249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.851218    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-dshm\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.851391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.851258    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c02de27-f10b-4aff-80de-228105a458c2-tls-certs\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.851391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.851300    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-home\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.851391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.851314    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-model-cache\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.851391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.851336    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-tmp-dir\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.851391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.851357    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxjn\" (UniqueName: \"kubernetes.io/projected/6c02de27-f10b-4aff-80de-228105a458c2-kube-api-access-lhxjn\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.851703 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.851664    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-home\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.851745 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.851726    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-model-cache\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.851799 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.851761    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-tmp-dir\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.853400 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.853376    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-dshm\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.853642 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.853627    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c02de27-f10b-4aff-80de-228105a458c2-tls-certs\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:52.859316 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:52.859298    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxjn\" (UniqueName: \"kubernetes.io/projected/6c02de27-f10b-4aff-80de-228105a458c2-kube-api-access-lhxjn\") pod \"e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") " pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:53.005556 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:53.005519    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:45:53.274064 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:53.274039    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"]
Sep 09 14:45:53.276699 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:45:53.276670    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c02de27_f10b_4aff_80de_228105a458c2.slice/crio-a3c2e5e472feddd941c80eadb68a743db21137a9919ea9956d08343d1d670c67 WatchSource:0}: Error finding container a3c2e5e472feddd941c80eadb68a743db21137a9919ea9956d08343d1d670c67: Status 404 returned error can't find the container with id a3c2e5e472feddd941c80eadb68a743db21137a9919ea9956d08343d1d670c67
Sep 09 14:45:53.326150 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:53.326126    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx" event={"ID":"6c02de27-f10b-4aff-80de-228105a458c2","Type":"ContainerStarted","Data":"a3c2e5e472feddd941c80eadb68a743db21137a9919ea9956d08343d1d670c67"}
Sep 09 14:45:54.332509 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:54.332476    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx" event={"ID":"6c02de27-f10b-4aff-80de-228105a458c2","Type":"ContainerStarted","Data":"ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210"}
Sep 09 14:45:54.349216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:45:54.349169    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx" podStartSLOduration=2.349157002 podStartE2EDuration="2.349157002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:45:54.347938906 +0000 UTC m=+2088.110734159" watchObservedRunningTime="2026-09-09 14:45:54.349157002 +0000 UTC m=+2088.111952241"
Sep 09 14:46:03.005979 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:03.005943    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:46:03.006399 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:03.005999    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:46:03.018363 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:03.018333    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:46:03.395822 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:03.395735    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:46:06.844202 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:06.844175    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:46:06.855205 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:06.855172    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pxnxs_8fae84b9-0f9a-4162-84b2-f3488a0e04af/ovn-acl-logging/0.log"
Sep 09 14:46:10.199450 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.199418    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"]
Sep 09 14:46:10.203559 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.203536    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.205755 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.205733    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e34b3797-kserve-self-signed-certs\""
Sep 09 14:46:10.213071 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.213048    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"]
Sep 09 14:46:10.299164 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.299134    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-tmp-dir\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.299302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.299191    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-dshm\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.299302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.299214    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-model-cache\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.299302 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.299283    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfp6\" (UniqueName: \"kubernetes.io/projected/b65b9b99-0b39-40be-ae28-cde4112d79a8-kube-api-access-kqfp6\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.299473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.299307    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-home\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.299473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.299408    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b65b9b99-0b39-40be-ae28-cde4112d79a8-tls-certs\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.400122 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.400073    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfp6\" (UniqueName: \"kubernetes.io/projected/b65b9b99-0b39-40be-ae28-cde4112d79a8-kube-api-access-kqfp6\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.400248 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.400129    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-home\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.400248 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.400203    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b65b9b99-0b39-40be-ae28-cde4112d79a8-tls-certs\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.400366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.400250    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-tmp-dir\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.400366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.400299    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-dshm\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.400366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.400324    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-model-cache\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.400604 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.400585    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-home\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.400646 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.400601    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-model-cache\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.400685 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.400655    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-tmp-dir\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.402389 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.402371    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-dshm\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.402637 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.402620    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b65b9b99-0b39-40be-ae28-cde4112d79a8-tls-certs\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.408262 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.408241    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfp6\" (UniqueName: \"kubernetes.io/projected/b65b9b99-0b39-40be-ae28-cde4112d79a8-kube-api-access-kqfp6\") pod \"e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") " pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.514216 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.514194    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:10.810239 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:10.810124    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"]
Sep 09 14:46:10.814176 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:46:10.814143    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb65b9b99_0b39_40be_ae28_cde4112d79a8.slice/crio-eab13c909f3eca168b594fb3e7aa5a638a134db2943fb13f958d14596fad6f9a WatchSource:0}: Error finding container eab13c909f3eca168b594fb3e7aa5a638a134db2943fb13f958d14596fad6f9a: Status 404 returned error can't find the container with id eab13c909f3eca168b594fb3e7aa5a638a134db2943fb13f958d14596fad6f9a
Sep 09 14:46:11.426277 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:11.426247    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r" event={"ID":"b65b9b99-0b39-40be-ae28-cde4112d79a8","Type":"ContainerStarted","Data":"d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e"}
Sep 09 14:46:11.426628 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:11.426286    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r" event={"ID":"b65b9b99-0b39-40be-ae28-cde4112d79a8","Type":"ContainerStarted","Data":"eab13c909f3eca168b594fb3e7aa5a638a134db2943fb13f958d14596fad6f9a"}
Sep 09 14:46:11.446150 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:11.446084    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r" podStartSLOduration=1.446068811 podStartE2EDuration="1.446068811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:46:11.442947631 +0000 UTC m=+2105.205742992" watchObservedRunningTime="2026-09-09 14:46:11.446068811 +0000 UTC m=+2105.208864051"
Sep 09 14:46:20.515260 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:20.515230    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:20.515625 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:20.515270    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:20.527863 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:20.527839    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:21.485763 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:21.485735    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:46:47.997193 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:47.997159    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"]
Sep 09 14:46:48.894184 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:46:48.894148    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"]
Sep 09 14:47:03.016729 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.016679    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx" podUID="6c02de27-f10b-4aff-80de-228105a458c2" containerName="main" containerID="cri-o://ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210" gracePeriod=45
Sep 09 14:47:03.262155 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.262130    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:47:03.386606 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.386534    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-tmp-dir\") pod \"6c02de27-f10b-4aff-80de-228105a458c2\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") "
Sep 09 14:47:03.386752 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.386607    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c02de27-f10b-4aff-80de-228105a458c2-tls-certs\") pod \"6c02de27-f10b-4aff-80de-228105a458c2\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") "
Sep 09 14:47:03.386752 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.386651    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-dshm\") pod \"6c02de27-f10b-4aff-80de-228105a458c2\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") "
Sep 09 14:47:03.386752 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.386709    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-model-cache\") pod \"6c02de27-f10b-4aff-80de-228105a458c2\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") "
Sep 09 14:47:03.386924 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.386752    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-home\") pod \"6c02de27-f10b-4aff-80de-228105a458c2\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") "
Sep 09 14:47:03.386924 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.386798    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhxjn\" (UniqueName: \"kubernetes.io/projected/6c02de27-f10b-4aff-80de-228105a458c2-kube-api-access-lhxjn\") pod \"6c02de27-f10b-4aff-80de-228105a458c2\" (UID: \"6c02de27-f10b-4aff-80de-228105a458c2\") "
Sep 09 14:47:03.386924 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.386816    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c02de27-f10b-4aff-80de-228105a458c2" (UID: "6c02de27-f10b-4aff-80de-228105a458c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:47:03.387082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.386973    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-home" (OuterVolumeSpecName: "home") pod "6c02de27-f10b-4aff-80de-228105a458c2" (UID: "6c02de27-f10b-4aff-80de-228105a458c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:47:03.387082 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.386984    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c02de27-f10b-4aff-80de-228105a458c2" (UID: "6c02de27-f10b-4aff-80de-228105a458c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:47:03.387197 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.387092    2532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-model-cache\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:03.387197 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.387125    2532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-home\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:03.387197 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.387155    2532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-tmp-dir\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:03.388966 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.388937    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c02de27-f10b-4aff-80de-228105a458c2-kube-api-access-lhxjn" (OuterVolumeSpecName: "kube-api-access-lhxjn") pod "6c02de27-f10b-4aff-80de-228105a458c2" (UID: "6c02de27-f10b-4aff-80de-228105a458c2"). InnerVolumeSpecName "kube-api-access-lhxjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:47:03.389206 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.389184    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-dshm" (OuterVolumeSpecName: "dshm") pod "6c02de27-f10b-4aff-80de-228105a458c2" (UID: "6c02de27-f10b-4aff-80de-228105a458c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:47:03.389336 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.389213    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c02de27-f10b-4aff-80de-228105a458c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c02de27-f10b-4aff-80de-228105a458c2" (UID: "6c02de27-f10b-4aff-80de-228105a458c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:47:03.488416 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.488386    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhxjn\" (UniqueName: \"kubernetes.io/projected/6c02de27-f10b-4aff-80de-228105a458c2-kube-api-access-lhxjn\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:03.488416 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.488412    2532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c02de27-f10b-4aff-80de-228105a458c2-tls-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:03.488416 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.488422    2532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c02de27-f10b-4aff-80de-228105a458c2-dshm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:03.686105 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.686011    2532 generic.go:356] "Generic (PLEG): container finished" podID="6c02de27-f10b-4aff-80de-228105a458c2" containerID="ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210" exitCode=0
Sep 09 14:47:03.686105 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.686065    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"
Sep 09 14:47:03.686281 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.686113    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx" event={"ID":"6c02de27-f10b-4aff-80de-228105a458c2","Type":"ContainerDied","Data":"ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210"}
Sep 09 14:47:03.686281 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.686164    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx" event={"ID":"6c02de27-f10b-4aff-80de-228105a458c2","Type":"ContainerDied","Data":"a3c2e5e472feddd941c80eadb68a743db21137a9919ea9956d08343d1d670c67"}
Sep 09 14:47:03.686281 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.686186    2532 scope.go:117] "RemoveContainer" containerID="ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210"
Sep 09 14:47:03.695581 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.695561    2532 scope.go:117] "RemoveContainer" containerID="ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210"
Sep 09 14:47:03.695886 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:47:03.695865    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210\": container with ID starting with ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210 not found: ID does not exist" containerID="ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210"
Sep 09 14:47:03.695989 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.695892    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210"} err="failed to get container status \"ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210\": rpc error: code = NotFound desc = could not find container \"ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210\": container with ID starting with ef215f6d6024691b824f7ea2d78a1d1b9898b4a7829c14e4b4d1bee809174210 not found: ID does not exist"
Sep 09 14:47:03.706904 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.706882    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"]
Sep 09 14:47:03.711008 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.710987    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e34b3797-kserve-5f5b87f899-7f6rx"]
Sep 09 14:47:03.913895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:03.913861    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r" podUID="b65b9b99-0b39-40be-ae28-cde4112d79a8" containerName="main" containerID="cri-o://d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e" gracePeriod=45
Sep 09 14:47:04.160904 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.160883    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:47:04.295560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.295480    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b65b9b99-0b39-40be-ae28-cde4112d79a8-tls-certs\") pod \"b65b9b99-0b39-40be-ae28-cde4112d79a8\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") "
Sep 09 14:47:04.295560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.295523    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-tmp-dir\") pod \"b65b9b99-0b39-40be-ae28-cde4112d79a8\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") "
Sep 09 14:47:04.295780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.295566    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-model-cache\") pod \"b65b9b99-0b39-40be-ae28-cde4112d79a8\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") "
Sep 09 14:47:04.295780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.295581    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-home\") pod \"b65b9b99-0b39-40be-ae28-cde4112d79a8\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") "
Sep 09 14:47:04.295780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.295610    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqfp6\" (UniqueName: \"kubernetes.io/projected/b65b9b99-0b39-40be-ae28-cde4112d79a8-kube-api-access-kqfp6\") pod \"b65b9b99-0b39-40be-ae28-cde4112d79a8\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") "
Sep 09 14:47:04.295780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.295634    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-dshm\") pod \"b65b9b99-0b39-40be-ae28-cde4112d79a8\" (UID: \"b65b9b99-0b39-40be-ae28-cde4112d79a8\") "
Sep 09 14:47:04.295995 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.295812    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "b65b9b99-0b39-40be-ae28-cde4112d79a8" (UID: "b65b9b99-0b39-40be-ae28-cde4112d79a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:47:04.295995 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.295833    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b65b9b99-0b39-40be-ae28-cde4112d79a8" (UID: "b65b9b99-0b39-40be-ae28-cde4112d79a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:47:04.295995 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.295879    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-home" (OuterVolumeSpecName: "home") pod "b65b9b99-0b39-40be-ae28-cde4112d79a8" (UID: "b65b9b99-0b39-40be-ae28-cde4112d79a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:47:04.296175 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.296019    2532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-tmp-dir\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:04.296175 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.296041    2532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-model-cache\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:04.296175 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.296053    2532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-home\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:04.297734 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.297707    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-dshm" (OuterVolumeSpecName: "dshm") pod "b65b9b99-0b39-40be-ae28-cde4112d79a8" (UID: "b65b9b99-0b39-40be-ae28-cde4112d79a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:47:04.297842 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.297729    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65b9b99-0b39-40be-ae28-cde4112d79a8-kube-api-access-kqfp6" (OuterVolumeSpecName: "kube-api-access-kqfp6") pod "b65b9b99-0b39-40be-ae28-cde4112d79a8" (UID: "b65b9b99-0b39-40be-ae28-cde4112d79a8"). InnerVolumeSpecName "kube-api-access-kqfp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:47:04.297842 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.297754    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b65b9b99-0b39-40be-ae28-cde4112d79a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b65b9b99-0b39-40be-ae28-cde4112d79a8" (UID: "b65b9b99-0b39-40be-ae28-cde4112d79a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:47:04.397347 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.397319    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqfp6\" (UniqueName: \"kubernetes.io/projected/b65b9b99-0b39-40be-ae28-cde4112d79a8-kube-api-access-kqfp6\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:04.397347 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.397344    2532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b65b9b99-0b39-40be-ae28-cde4112d79a8-dshm\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:04.397505 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.397353    2532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b65b9b99-0b39-40be-ae28-cde4112d79a8-tls-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:04.692974 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.692890    2532 generic.go:356] "Generic (PLEG): container finished" podID="b65b9b99-0b39-40be-ae28-cde4112d79a8" containerID="d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e" exitCode=0
Sep 09 14:47:04.692974 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.692933    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"
Sep 09 14:47:04.693186 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.692977    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r" event={"ID":"b65b9b99-0b39-40be-ae28-cde4112d79a8","Type":"ContainerDied","Data":"d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e"}
Sep 09 14:47:04.693186 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.693027    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r" event={"ID":"b65b9b99-0b39-40be-ae28-cde4112d79a8","Type":"ContainerDied","Data":"eab13c909f3eca168b594fb3e7aa5a638a134db2943fb13f958d14596fad6f9a"}
Sep 09 14:47:04.693186 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.693045    2532 scope.go:117] "RemoveContainer" containerID="d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e"
Sep 09 14:47:04.702241 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.702226    2532 scope.go:117] "RemoveContainer" containerID="d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e"
Sep 09 14:47:04.702517 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:47:04.702500    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e\": container with ID starting with d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e not found: ID does not exist" containerID="d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e"
Sep 09 14:47:04.702557 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.702525    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e"} err="failed to get container status \"d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e\": rpc error: code = NotFound desc = could not find container \"d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e\": container with ID starting with d46611918dafcea483cf39ccd04b41d2dbdae86dd8ccd0d30187f6a83224507e not found: ID does not exist"
Sep 09 14:47:04.713963 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.713941    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"]
Sep 09 14:47:04.720141 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.720118    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e34b3797-kserve-75cdc8cf55-8xn8r"]
Sep 09 14:47:04.734104 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.734076    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c02de27-f10b-4aff-80de-228105a458c2" path="/var/lib/kubelet/pods/6c02de27-f10b-4aff-80de-228105a458c2/volumes"
Sep 09 14:47:04.734500 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:04.734487    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65b9b99-0b39-40be-ae28-cde4112d79a8" path="/var/lib/kubelet/pods/b65b9b99-0b39-40be-ae28-cde4112d79a8/volumes"
Sep 09 14:47:07.011661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.011631    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"]
Sep 09 14:47:07.012221 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.012201    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65b9b99-0b39-40be-ae28-cde4112d79a8" containerName="main"
Sep 09 14:47:07.012310 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.012223    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65b9b99-0b39-40be-ae28-cde4112d79a8" containerName="main"
Sep 09 14:47:07.012310 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.012272    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c02de27-f10b-4aff-80de-228105a458c2" containerName="main"
Sep 09 14:47:07.012310 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.012283    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c02de27-f10b-4aff-80de-228105a458c2" containerName="main"
Sep 09 14:47:07.012465 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.012400    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c02de27-f10b-4aff-80de-228105a458c2" containerName="main"
Sep 09 14:47:07.012465 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.012414    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65b9b99-0b39-40be-ae28-cde4112d79a8" containerName="main"
Sep 09 14:47:07.016775 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.016752    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.026613 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.026544    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"]
Sep 09 14:47:07.119135 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.119082    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhhvx\" (UniqueName: \"kubernetes.io/projected/9fc7aff5-28af-4760-a9fc-85762db43196-kube-api-access-qhhvx\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.119238 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.119150    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fc7aff5-28af-4760-a9fc-85762db43196-maas-api-tls\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.119238 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.119203    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fc7aff5-28af-4760-a9fc-85762db43196-metrics-certs\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.119329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.119245    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7aff5-28af-4760-a9fc-85762db43196-openshift-service-ca\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.219863 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.219841    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7aff5-28af-4760-a9fc-85762db43196-openshift-service-ca\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.219955 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.219918    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhhvx\" (UniqueName: \"kubernetes.io/projected/9fc7aff5-28af-4760-a9fc-85762db43196-kube-api-access-qhhvx\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.219955 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.219943    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fc7aff5-28af-4760-a9fc-85762db43196-maas-api-tls\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.220031 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.219962    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fc7aff5-28af-4760-a9fc-85762db43196-metrics-certs\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.220546 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.220526    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7aff5-28af-4760-a9fc-85762db43196-openshift-service-ca\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.222244 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.222218    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fc7aff5-28af-4760-a9fc-85762db43196-metrics-certs\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.222355 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.222336    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fc7aff5-28af-4760-a9fc-85762db43196-maas-api-tls\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.227465 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.227444    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhhvx\" (UniqueName: \"kubernetes.io/projected/9fc7aff5-28af-4760-a9fc-85762db43196-kube-api-access-qhhvx\") pod \"maas-api-6fcbb6857c-j2d5g\" (UID: \"9fc7aff5-28af-4760-a9fc-85762db43196\") " pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.329059 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.329001    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:07.602421 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.602397    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"]
Sep 09 14:47:07.604569 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:47:07.604542    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc7aff5_28af_4760_a9fc_85762db43196.slice/crio-37794ae3fb21125d6ce739293f813d72e4324d839a741f5125f4ff087a9cd3eb WatchSource:0}: Error finding container 37794ae3fb21125d6ce739293f813d72e4324d839a741f5125f4ff087a9cd3eb: Status 404 returned error can't find the container with id 37794ae3fb21125d6ce739293f813d72e4324d839a741f5125f4ff087a9cd3eb
Sep 09 14:47:07.605887 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.605870    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 14:47:07.709780 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:07.709742    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g" event={"ID":"9fc7aff5-28af-4760-a9fc-85762db43196","Type":"ContainerStarted","Data":"37794ae3fb21125d6ce739293f813d72e4324d839a741f5125f4ff087a9cd3eb"}
Sep 09 14:47:10.726763 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:10.726723    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g" event={"ID":"9fc7aff5-28af-4760-a9fc-85762db43196","Type":"ContainerStarted","Data":"81edd9935e0c292e0c30020a44d54941611316c1875e16c768c154cce9460600"}
Sep 09 14:47:10.727208 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:10.726870    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:10.747296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:10.747250    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g" podStartSLOduration=2.471408087 podStartE2EDuration="4.747236216s" totalImagesPullingTime="2.275828129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:47:10.745915617 +0000 UTC m=+2164.508710889" watchObservedRunningTime="2026-09-09 14:47:10.747236216 +0000 UTC m=+2164.510031459"
Sep 09 14:47:16.737452 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:16.737426    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6fcbb6857c-j2d5g"
Sep 09 14:47:16.784235 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:16.784200    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"]
Sep 09 14:47:16.784520 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:16.784458    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn" podUID="dfaa9dee-1da4-4a40-a010-56391d76380b" containerName="maas-api" containerID="cri-o://64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e" gracePeriod=30
Sep 09 14:47:18.028376 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.028348    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:47:18.100895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.100825    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-maas-api-tls\") pod \"dfaa9dee-1da4-4a40-a010-56391d76380b\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") "
Sep 09 14:47:18.101008 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.100906    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pzj5\" (UniqueName: \"kubernetes.io/projected/dfaa9dee-1da4-4a40-a010-56391d76380b-kube-api-access-9pzj5\") pod \"dfaa9dee-1da4-4a40-a010-56391d76380b\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") "
Sep 09 14:47:18.101008 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.100948    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-metrics-certs\") pod \"dfaa9dee-1da4-4a40-a010-56391d76380b\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") "
Sep 09 14:47:18.101113 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.101022    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaa9dee-1da4-4a40-a010-56391d76380b-openshift-service-ca\") pod \"dfaa9dee-1da4-4a40-a010-56391d76380b\" (UID: \"dfaa9dee-1da4-4a40-a010-56391d76380b\") "
Sep 09 14:47:18.101480 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.101442    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfaa9dee-1da4-4a40-a010-56391d76380b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfaa9dee-1da4-4a40-a010-56391d76380b" (UID: "dfaa9dee-1da4-4a40-a010-56391d76380b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 14:47:18.102739 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.102717    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfaa9dee-1da4-4a40-a010-56391d76380b" (UID: "dfaa9dee-1da4-4a40-a010-56391d76380b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:47:18.102949 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.102929    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfaa9dee-1da4-4a40-a010-56391d76380b-kube-api-access-9pzj5" (OuterVolumeSpecName: "kube-api-access-9pzj5") pod "dfaa9dee-1da4-4a40-a010-56391d76380b" (UID: "dfaa9dee-1da4-4a40-a010-56391d76380b"). InnerVolumeSpecName "kube-api-access-9pzj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:47:18.103021 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.103006    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dfaa9dee-1da4-4a40-a010-56391d76380b" (UID: "dfaa9dee-1da4-4a40-a010-56391d76380b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:47:18.202134 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.202089    2532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaa9dee-1da4-4a40-a010-56391d76380b-openshift-service-ca\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:18.202134 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.202135    2532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-maas-api-tls\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:18.202257 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.202144    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pzj5\" (UniqueName: \"kubernetes.io/projected/dfaa9dee-1da4-4a40-a010-56391d76380b-kube-api-access-9pzj5\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:18.202257 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.202153    2532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfaa9dee-1da4-4a40-a010-56391d76380b-metrics-certs\") on node \"ip-10-0-139-251.ec2.internal\" DevicePath \"\""
Sep 09 14:47:18.764606 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.764572    2532 generic.go:356] "Generic (PLEG): container finished" podID="dfaa9dee-1da4-4a40-a010-56391d76380b" containerID="64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e" exitCode=0
Sep 09 14:47:18.764710 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.764614    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"
Sep 09 14:47:18.764710 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.764630    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn" event={"ID":"dfaa9dee-1da4-4a40-a010-56391d76380b","Type":"ContainerDied","Data":"64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e"}
Sep 09 14:47:18.764710 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.764673    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn" event={"ID":"dfaa9dee-1da4-4a40-a010-56391d76380b","Type":"ContainerDied","Data":"e6e6a705ed5da3618d14d9c145687147d66fafe297221762b0537317459752de"}
Sep 09 14:47:18.764710 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.764696    2532 scope.go:117] "RemoveContainer" containerID="64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e"
Sep 09 14:47:18.773361 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.773344    2532 scope.go:117] "RemoveContainer" containerID="64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e"
Sep 09 14:47:18.773633 ip-10-0-139-251 kubenswrapper[2532]: E0909 14:47:18.773616    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e\": container with ID starting with 64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e not found: ID does not exist" containerID="64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e"
Sep 09 14:47:18.773681 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.773642    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e"} err="failed to get container status \"64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e\": rpc error: code = NotFound desc = could not find container \"64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e\": container with ID starting with 64ce42fa5bc26fc89ad856180c83604c7cdfc485453b78bb7eb2d4f8d8a6ef3e not found: ID does not exist"
Sep 09 14:47:18.781138 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.781111    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"]
Sep 09 14:47:18.785409 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:18.785389    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cbb79749-n8lpn"]
Sep 09 14:47:20.735004 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:20.734972    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfaa9dee-1da4-4a40-a010-56391d76380b" path="/var/lib/kubelet/pods/dfaa9dee-1da4-4a40-a010-56391d76380b/volumes"
Sep 09 14:47:43.134563 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:43.134532    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-686db84dc8-wkg7l_0c159d6b-25ae-4b67-8596-64198ca7e225/copy-manifests/0.log"
Sep 09 14:47:43.152798 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:43.152773    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-686db84dc8-wkg7l_0c159d6b-25ae-4b67-8596-64198ca7e225/manager/0.log"
Sep 09 14:47:43.506683 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:43.506657    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6fcbb6857c-j2d5g_9fc7aff5-28af-4760-a9fc-85762db43196/maas-api/0.log"
Sep 09 14:47:43.989111 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:43.989049    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-fv79k_d1ec597b-328b-4a1a-8f47-dcb3bd1440a6/postgres/0.log"
Sep 09 14:47:45.142667 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:45.142635    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-pp66l_e027b5d5-e827-4e21-9ccf-0070914e9011/kuadrant-console-plugin/0.log"
Sep 09 14:47:46.079787 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:46.079756    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd84d6f45-t56sb_a8938794-daba-4785-b9d2-5b8d9870e6a8/kube-auth-proxy/0.log"
Sep 09 14:47:46.441952 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:46.441842    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-m9gkd_2301e4ac-b43b-44f5-8b90-4dffed7fa817/payload-pre-processing/0.log"
Sep 09 14:47:47.350576 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:47.350533    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-2b2ms_c4af6929-882f-4325-ab65-022e8ae15afe/main/0.log"
Sep 09 14:47:47.460473 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:47.460446    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxsnsz_c52f20f6-a72a-435f-9a8f-6109ac8bd26a/main/0.log"
Sep 09 14:47:47.693715 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:47.693629    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-7hpmg_57935d16-670b-4427-b455-449aded29fc1/main/0.log"
Sep 09 14:47:56.305848 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:56.305818    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8xz8x_04d39c37-e950-4def-86cf-bc268ada8e3f/global-pull-secret-syncer/0.log"
Sep 09 14:47:56.457063 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:56.457026    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-svpv6_49ea2ecc-a574-4969-85d4-1c6419ce4a06/konnectivity-agent/0.log"
Sep 09 14:47:56.482438 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:47:56.482413    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-251.ec2.internal_d6aac1761331084f3b830a859c99fcd9/haproxy/0.log"
Sep 09 14:48:01.603946 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:01.603913    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-pp66l_e027b5d5-e827-4e21-9ccf-0070914e9011/kuadrant-console-plugin/0.log"
Sep 09 14:48:03.405731 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.405645    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6ecf7ca-7459-4e4c-8377-d3a1696377ed/alertmanager/0.log"
Sep 09 14:48:03.437391 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.437355    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6ecf7ca-7459-4e4c-8377-d3a1696377ed/config-reloader/0.log"
Sep 09 14:48:03.457781 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.457749    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6ecf7ca-7459-4e4c-8377-d3a1696377ed/kube-rbac-proxy-web/0.log"
Sep 09 14:48:03.477217 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.477188    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6ecf7ca-7459-4e4c-8377-d3a1696377ed/kube-rbac-proxy/0.log"
Sep 09 14:48:03.500694 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.500668    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6ecf7ca-7459-4e4c-8377-d3a1696377ed/kube-rbac-proxy-metric/0.log"
Sep 09 14:48:03.520714 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.520688    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6ecf7ca-7459-4e4c-8377-d3a1696377ed/prom-label-proxy/0.log"
Sep 09 14:48:03.544494 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.544470    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6ecf7ca-7459-4e4c-8377-d3a1696377ed/init-config-reloader/0.log"
Sep 09 14:48:03.612149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.612122    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-25xwv_4ae0ea2c-ab1f-469c-a331-bfc1cab73317/kube-state-metrics/0.log"
Sep 09 14:48:03.626965 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.626941    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-25xwv_4ae0ea2c-ab1f-469c-a331-bfc1cab73317/kube-rbac-proxy-main/0.log"
Sep 09 14:48:03.649484 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.649461    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-25xwv_4ae0ea2c-ab1f-469c-a331-bfc1cab73317/kube-rbac-proxy-self/0.log"
Sep 09 14:48:03.673721 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.673660    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f49bd8f8-zx26s_7a9e9fb2-cb35-4732-b75d-4db5e66b3f27/metrics-server/0.log"
Sep 09 14:48:03.714235 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.714211    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-jw8nl_a68a09d4-7a18-44db-bcbc-3e2e3946c5f7/monitoring-plugin/0.log"
Sep 09 14:48:03.891942 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.891919    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s7ng9_38c179f5-4f8d-4491-a4cd-ceafdc25b295/node-exporter/0.log"
Sep 09 14:48:03.912779 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.912709    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s7ng9_38c179f5-4f8d-4491-a4cd-ceafdc25b295/kube-rbac-proxy/0.log"
Sep 09 14:48:03.937171 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.937112    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s7ng9_38c179f5-4f8d-4491-a4cd-ceafdc25b295/init-textfile/0.log"
Sep 09 14:48:03.962941 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.962922    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-tnrxq_24a655ce-1c8e-45fb-97cb-00bc12e09e20/kube-rbac-proxy-main/0.log"
Sep 09 14:48:03.981437 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:03.981418    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-tnrxq_24a655ce-1c8e-45fb-97cb-00bc12e09e20/kube-rbac-proxy-self/0.log"
Sep 09 14:48:04.005855 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.005830    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-tnrxq_24a655ce-1c8e-45fb-97cb-00bc12e09e20/openshift-state-metrics/0.log"
Sep 09 14:48:04.042041 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.042019    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40cabd6a-194f-472a-a714-1bb0b495f954/prometheus/0.log"
Sep 09 14:48:04.060125 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.060088    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40cabd6a-194f-472a-a714-1bb0b495f954/config-reloader/0.log"
Sep 09 14:48:04.085464 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.085441    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40cabd6a-194f-472a-a714-1bb0b495f954/thanos-sidecar/0.log"
Sep 09 14:48:04.105212 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.105191    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40cabd6a-194f-472a-a714-1bb0b495f954/kube-rbac-proxy-web/0.log"
Sep 09 14:48:04.123362 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.123345    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40cabd6a-194f-472a-a714-1bb0b495f954/kube-rbac-proxy/0.log"
Sep 09 14:48:04.142138 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.142090    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40cabd6a-194f-472a-a714-1bb0b495f954/kube-rbac-proxy-thanos/0.log"
Sep 09 14:48:04.161676 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.161658    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40cabd6a-194f-472a-a714-1bb0b495f954/init-config-reloader/0.log"
Sep 09 14:48:04.186818 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.186797    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-fhwz7_8455845f-c049-4206-ac60-a4cf28fd903f/prometheus-operator/0.log"
Sep 09 14:48:04.202359 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.202308    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-fhwz7_8455845f-c049-4206-ac60-a4cf28fd903f/kube-rbac-proxy/0.log"
Sep 09 14:48:04.251195 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.251174    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d859d47f9-4pk4c_b25de1ba-24ef-4c06-8cba-957b49915805/telemeter-client/0.log"
Sep 09 14:48:04.269366 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.269347    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d859d47f9-4pk4c_b25de1ba-24ef-4c06-8cba-957b49915805/reload/0.log"
Sep 09 14:48:04.287938 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.287917    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d859d47f9-4pk4c_b25de1ba-24ef-4c06-8cba-957b49915805/kube-rbac-proxy/0.log"
Sep 09 14:48:04.316091 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.316076    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58578cccb4-8jclj_bd26eb7e-40ac-4286-9ff7-6a2dead1d728/thanos-query/0.log"
Sep 09 14:48:04.333952 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.333935    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58578cccb4-8jclj_bd26eb7e-40ac-4286-9ff7-6a2dead1d728/kube-rbac-proxy-web/0.log"
Sep 09 14:48:04.351661 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.351641    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58578cccb4-8jclj_bd26eb7e-40ac-4286-9ff7-6a2dead1d728/kube-rbac-proxy/0.log"
Sep 09 14:48:04.371452 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.371435    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58578cccb4-8jclj_bd26eb7e-40ac-4286-9ff7-6a2dead1d728/prom-label-proxy/0.log"
Sep 09 14:48:04.391844 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.391818    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58578cccb4-8jclj_bd26eb7e-40ac-4286-9ff7-6a2dead1d728/kube-rbac-proxy-rules/0.log"
Sep 09 14:48:04.411247 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:04.411225    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58578cccb4-8jclj_bd26eb7e-40ac-4286-9ff7-6a2dead1d728/kube-rbac-proxy-metrics/0.log"
Sep 09 14:48:05.841690 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:05.841656    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"]
Sep 09 14:48:05.842149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:05.842115    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfaa9dee-1da4-4a40-a010-56391d76380b" containerName="maas-api"
Sep 09 14:48:05.842149 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:05.842130    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfaa9dee-1da4-4a40-a010-56391d76380b" containerName="maas-api"
Sep 09 14:48:05.842279 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:05.842260    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfaa9dee-1da4-4a40-a010-56391d76380b" containerName="maas-api"
Sep 09 14:48:05.849240 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:05.846204    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:05.849376 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:05.849293    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tq4f9\"/\"default-dockercfg-4q82f\""
Sep 09 14:48:05.864369 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:05.864336    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tq4f9\"/\"kube-root-ca.crt\""
Sep 09 14:48:05.864494 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:05.864377    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tq4f9\"/\"openshift-service-ca.crt\""
Sep 09 14:48:05.866462 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:05.866439    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"]
Sep 09 14:48:06.001988 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.001957    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-podres\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.002146 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.001996    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-proc\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.002146 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.002025    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-lib-modules\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.002238 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.002149    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-sys\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.002238 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.002182    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdtl2\" (UniqueName: \"kubernetes.io/projected/1d966881-8857-4dd3-a12c-4fe2f9cf5749-kube-api-access-jdtl2\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.103519 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.103453    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtl2\" (UniqueName: \"kubernetes.io/projected/1d966881-8857-4dd3-a12c-4fe2f9cf5749-kube-api-access-jdtl2\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.103519 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.103501    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-podres\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.103672 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.103621    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-podres\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.103713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.103670    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-proc\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.103713 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.103708    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-lib-modules\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.103779 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.103754    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-proc\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.103814 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.103779    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-sys\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.103875 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.103862    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-sys\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.103912 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.103879    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d966881-8857-4dd3-a12c-4fe2f9cf5749-lib-modules\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.111196 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.111170    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdtl2\" (UniqueName: \"kubernetes.io/projected/1d966881-8857-4dd3-a12c-4fe2f9cf5749-kube-api-access-jdtl2\") pod \"perf-node-gather-daemonset-mxr5n\" (UID: \"1d966881-8857-4dd3-a12c-4fe2f9cf5749\") " pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.159676 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.159654    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.427205 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.427174    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"]
Sep 09 14:48:06.429072 ip-10-0-139-251 kubenswrapper[2532]: W0909 14:48:06.429047    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1d966881_8857_4dd3_a12c_4fe2f9cf5749.slice/crio-c21c1b5d5f58c1520054aa7a0815eb0bfb9d2446a08410698365fd210041cad6 WatchSource:0}: Error finding container c21c1b5d5f58c1520054aa7a0815eb0bfb9d2446a08410698365fd210041cad6: Status 404 returned error can't find the container with id c21c1b5d5f58c1520054aa7a0815eb0bfb9d2446a08410698365fd210041cad6
Sep 09 14:48:06.731720 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.731695    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7fff8bd786-thld2_63887fb4-3b5d-43ec-ab33-36f8fb16ea03/console/0.log"
Sep 09 14:48:06.989595 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.989519    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n" event={"ID":"1d966881-8857-4dd3-a12c-4fe2f9cf5749","Type":"ContainerStarted","Data":"ef1a42ded8072cec9d30ebb3832ec0ca8f6154214329853600aaaba7865d5fb3"}
Sep 09 14:48:06.989595 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.989586    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:06.989953 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:06.989615    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n" event={"ID":"1d966881-8857-4dd3-a12c-4fe2f9cf5749","Type":"ContainerStarted","Data":"c21c1b5d5f58c1520054aa7a0815eb0bfb9d2446a08410698365fd210041cad6"}
Sep 09 14:48:07.006643 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:07.006598    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n" podStartSLOduration=2.00658251 podStartE2EDuration="2.00658251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:48:07.00437471 +0000 UTC m=+2220.767169951" watchObservedRunningTime="2026-09-09 14:48:07.00658251 +0000 UTC m=+2220.769377749"
Sep 09 14:48:08.550314 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:08.550281    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8b6ld_82696d15-9d8e-4912-9084-255e8fac69a4/dns/0.log"
Sep 09 14:48:08.582264 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:08.582238    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8b6ld_82696d15-9d8e-4912-9084-255e8fac69a4/kube-rbac-proxy/0.log"
Sep 09 14:48:08.616898 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:08.616881    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cn577_be545f7f-c99c-47c4-9e0e-1e690afab968/dns-node-resolver/0.log"
Sep 09 14:48:09.234420 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:09.234392    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4z47c_5044a9fc-f558-40d3-9525-f07cb8171534/node-ca/0.log"
Sep 09 14:48:10.461106 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:10.461080    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd84d6f45-t56sb_a8938794-daba-4785-b9d2-5b8d9870e6a8/kube-auth-proxy/0.log"
Sep 09 14:48:10.583560 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:10.583532    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-m9gkd_2301e4ac-b43b-44f5-8b90-4dffed7fa817/payload-pre-processing/0.log"
Sep 09 14:48:11.504365 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:11.504331    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vxgj6_0f63aa86-01bb-4492-b0bd-ed04bf50908f/serve-healthcheck-canary/0.log"
Sep 09 14:48:11.982537 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:11.982509    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4wch_d9e2b2bb-425b-45a7-9213-67cc4f1e626e/kube-rbac-proxy/0.log"
Sep 09 14:48:12.018296 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:12.018270    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4wch_d9e2b2bb-425b-45a7-9213-67cc4f1e626e/exporter/0.log"
Sep 09 14:48:12.082205 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:12.082178    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4wch_d9e2b2bb-425b-45a7-9213-67cc4f1e626e/extractor/0.log"
Sep 09 14:48:13.004249 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:13.004223    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tq4f9/perf-node-gather-daemonset-mxr5n"
Sep 09 14:48:15.068895 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:15.068822    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-686db84dc8-wkg7l_0c159d6b-25ae-4b67-8596-64198ca7e225/manager/0.log"
Sep 09 14:48:15.087329 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:15.087308    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-686db84dc8-wkg7l_0c159d6b-25ae-4b67-8596-64198ca7e225/copy-manifests/0.log"
Sep 09 14:48:16.366370 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:16.366345    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-784f875f86-7g92d_5f27ee2a-4c63-4595-850f-bb657c264115/manager/0.log"
Sep 09 14:48:16.456951 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:16.456926    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd84d6f45-t56sb_a8938794-daba-4785-b9d2-5b8d9870e6a8/kube-auth-proxy/0.log"
Sep 09 14:48:16.544606 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:16.544570    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-m9gkd_2301e4ac-b43b-44f5-8b90-4dffed7fa817/payload-pre-processing/0.log"
Sep 09 14:48:18.236936 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:18.236906    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd84d6f45-t56sb_a8938794-daba-4785-b9d2-5b8d9870e6a8/kube-auth-proxy/0.log"
Sep 09 14:48:18.322505 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:18.322477    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-m9gkd_2301e4ac-b43b-44f5-8b90-4dffed7fa817/payload-pre-processing/0.log"
Sep 09 14:48:22.705584 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:22.705554    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5cc9_64100743-6529-42aa-8204-8138f81c29c7/kube-multus-additional-cni-plugins/0.log"
Sep 09 14:48:22.725993 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:22.725964    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5cc9_64100743-6529-42aa-8204-8138f81c29c7/egress-router-binary-copy/0.log"
Sep 09 14:48:22.746651 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:22.746630    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5cc9_64100743-6529-42aa-8204-8138f81c29c7/cni-plugins/0.log"
Sep 09 14:48:22.768159 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:22.768138    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5cc9_64100743-6529-42aa-8204-8138f81c29c7/bond-cni-plugin/0.log"
Sep 09 14:48:22.788817 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:22.788790    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5cc9_64100743-6529-42aa-8204-8138f81c29c7/routeoverride-cni/0.log"
Sep 09 14:48:22.806979 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:22.806955    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5cc9_64100743-6529-42aa-8204-8138f81c29c7/whereabouts-cni-bincopy/0.log"
Sep 09 14:48:22.824515 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:22.824494    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5cc9_64100743-6529-42aa-8204-8138f81c29c7/whereabouts-cni/0.log"
Sep 09 14:48:23.016565 ip-10-0-139-251 kubenswrapper[2532]: I0909 14:48:23.016541    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dghxg_bd746049-41ca-4402-93a6-320c0828fcbb/kube-multus/0.log"