Sep 11 18:13:35.567014 ip-10-0-139-186 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 18:13:35.567277 ip-10-0-139-186 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 18:13:35.567382 ip-10-0-139-186 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 18:13:35.567928 ip-10-0-139-186 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 18:13:45.587347 ip-10-0-139-186 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 18:13:45.587364 ip-10-0-139-186 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8d91ea661e01415fa56399baf9885ef4 --
Sep 11 18:15:35.295936 ip-10-0-139-186 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 18:15:35.750941 ip-10-0-139-186 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:35.750941 ip-10-0-139-186 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 18:15:35.750941 ip-10-0-139-186 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:35.750941 ip-10-0-139-186 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 18:15:35.750941 ip-10-0-139-186 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:35.751750 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.751662    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 18:15:35.755548 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755533    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:35.755548 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755549    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755554    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755557    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755561    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755564    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755566    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755569    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755572    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755575    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755578    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755581    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755587    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755589    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755592    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755594    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755597    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755600    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755603    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755606    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755608    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:35.755623 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755611    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755613    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755616    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755619    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755621    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755624    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755627    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755629    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755635    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755639    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755642    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755657    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755660    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755663    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755665    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755668    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755671    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755677    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755679    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:35.756112 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755682    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755686    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755689    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755692    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755695    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755697    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755700    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755702    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755705    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755707    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755710    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755712    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755715    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755718    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755720    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755723    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755726    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755729    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755732    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755734    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:35.756579 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755737    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755739    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755742    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755744    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755747    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755750    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755752    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755755    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755757    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755761    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755764    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755768    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755771    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755773    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755776    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755779    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755781    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755784    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755787    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755789    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:35.757074 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755792    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755794    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755797    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755799    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755801    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755804    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755806    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755810    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755813    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755816    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755819    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755822    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755824    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755827    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755829    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755832    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755836    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755839    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.755843    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:35.757549 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756584    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756591    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756597    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756600    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756603    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756607    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756610    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756613    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756615    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756618    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756621    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756624    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756626    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756629    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756631    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756634    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756636    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756639    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756641    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:35.758032 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756644    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756663    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756666    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756669    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756671    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756674    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756676    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756679    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756681    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756684    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756687    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756690    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756692    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756695    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756697    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756699    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756702    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756704    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756707    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756710    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:35.758506 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756713    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756716    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756719    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756722    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756724    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756727    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756729    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756732    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756734    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756737    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756740    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756744    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756747    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756750    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756753    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756756    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756759    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756761    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756764    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:35.759013 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756767    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756770    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756773    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756775    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756778    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756780    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756783    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756785    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756788    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756791    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756793    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756796    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756799    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756802    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756804    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756806    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756809    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756812    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756814    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756817    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:35.759498 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756819    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756822    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756824    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756827    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756829    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756832    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756835    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756837    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756840    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756842    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756845    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756848    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756850    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756856    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756858    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756863    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756866    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756869    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756873    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:35.760007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756876    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.756879    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757229    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757233    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757236    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757238    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757241    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757244    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757247    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757250    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757254    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757258    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757261    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757263    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757266    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757269    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757272    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757274    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757277    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:35.760459 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757279    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757282    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757284    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757288    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757291    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757294    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757297    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757301    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757304    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757306    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757309    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757312    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757315    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757318    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757320    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757323    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757326    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757328    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757330    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757333    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:35.760963 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757336    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757339    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757341    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757344    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757346    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757349    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757351    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757354    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757357    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757359    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757362    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757364    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757367    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757370    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757373    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757375    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757377    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757380    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757382    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757386    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:35.761453 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757388    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757391    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757394    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757396    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757400    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757402    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757405    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757407    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757410    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757412    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757415    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757417    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757420    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757423    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757426    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757428    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757430    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757433    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757435    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757438    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:35.761973 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757440    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757444    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757446    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757449    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757451    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757454    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757456    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757458    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757461    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757463    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757466    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757470    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757472    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757475    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757477    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757480    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757484    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757487    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757490    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757493    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:35.762490 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757495    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.757498    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757577    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757625    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757633    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757638    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757642    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757659    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757664    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757668    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757671    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757675    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757678    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757682    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757685    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757688    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757691    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757694    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757696    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757699    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757703    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757706    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757709    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757715    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 18:15:35.762984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757718    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757722    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757725    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757729    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757733    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757737    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757740    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757743    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757746    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757749    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757753    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757756    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757758    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757761    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757764    2578 flags.go:64] FLAG: --enable-server="true"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757768    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757772    2578 flags.go:64] FLAG: --event-burst="100"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757775    2578 flags.go:64] FLAG: --event-qps="50"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757778    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757781    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757784    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757788    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757790    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757793    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757796    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 11 18:15:35.763641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757799    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757802    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757805    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757808    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757811    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757814    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757818    2578 flags.go:64] FLAG: --feature-gates=""
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757822    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757825    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757828    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757831    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757835    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757838    2578 flags.go:64] FLAG: --help="false"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757841    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-139-186.ec2.internal"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757845    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757848    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757850    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757854    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757857    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757860    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757863    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757866    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757869    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 18:15:35.764249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757872    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757875    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757878    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757881    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757884    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757887    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757890    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757892    2578 flags.go:64] FLAG: --lock-file=""
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757895    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757898    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757901    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757907    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757909    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757912    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757915    2578 flags.go:64] FLAG: --logging-format="text"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757919    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757923    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757925    2578 flags.go:64] FLAG: --manifest-url=""
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757928    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757933    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757936    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757940    2578 flags.go:64] FLAG: --max-pods="110"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757944    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757947    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757950    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 18:15:35.765218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757952    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757955    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757958    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757961    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757968    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757971    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757974    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757977    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757981    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757986    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757989    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757992    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757995    2578 flags.go:64] FLAG: --port="10250"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.757998    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758001    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cf2302864d0de330"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758005    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758008    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758011    2578 flags.go:64] FLAG: --register-node="true"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758014    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758017    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758020    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758023    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758027    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758031    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758034    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 18:15:35.766285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758037    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758040    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758043    2578 flags.go:64] FLAG: --runonce="false"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758046    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758049    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758052    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758055    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758058    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758061    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758064    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758068    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758071    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758073    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758076    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758079    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758083    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758086    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758089    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758094    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758097    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758100    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758104    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758106    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758109    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758112    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 18:15:35.767393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758115    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758118    2578 flags.go:64] FLAG: --v="2"
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758121    2578 flags.go:64] FLAG: --version="false"
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758125    2578 flags.go:64] FLAG: --vmodule=""
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758129    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.758133    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758247    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758252    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758256    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758259    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758263    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758266    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758269    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758272    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758274    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758277    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758280    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758282    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758285    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758287    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758290    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:35.768348 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758293    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758295    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758298    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758301    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758304    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758306    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758309    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758312    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758314    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758317    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758319    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758322    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758325    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758327    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758330    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758333    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758336    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758338    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758341    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758344    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:35.769246 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758346    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758350    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758353    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758356    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758358    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758361    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758363    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758366    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758369    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758371    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758374    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758376    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758379    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758382    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758384    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758389    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758393    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758396    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758399    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:35.769955 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758402    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758405    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758408    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758411    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758413    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758416    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758419    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758422    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758424    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758427    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758429    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758432    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758434    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758437    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758442    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758444    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758447    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758450    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758452    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758455    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:35.770427 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758457    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758460    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758462    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758465    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758468    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758470    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758473    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758475    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758478    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758480    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758485    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758488    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758491    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758494    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758497    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758500    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758502    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758505    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758508    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758511    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:35.770947 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758514    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758516    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758519    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758522    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.758524    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.759041    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.766684    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.766709    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.766730    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.766740    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766825    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766833    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766838    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766843    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766848    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:35.771484 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766852    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766857    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766862    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766867    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766872    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766876    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766881    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766886    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766890    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766895    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766900    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766904    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766908    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766912    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766917    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766923    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766929    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766934    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766938    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:35.771876 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766944    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766950    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766955    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766959    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766963    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766969    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766974    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766978    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766982    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766986    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766991    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.766995    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767000    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767004    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767008    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767012    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767017    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767021    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767025    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:35.772335 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767029    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767033    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767037    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767042    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767046    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767050    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767054    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767060    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767066    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767071    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767076    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767081    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767085    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767089    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767094    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767098    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767102    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767106    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767112    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767118    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:35.772850 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767122    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767126    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767130    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767134    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767138    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767143    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767147    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767151    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767155    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767159    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767163    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767168    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767172    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767176    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767180    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767184    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767188    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767192    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767196    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767201    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:35.773353 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767205    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767209    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767214    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767218    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767222    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767226    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767230    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767234    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767239    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767243    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767247    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767251    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767256    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767260    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767264    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:35.773852 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767268    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.767276    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767453    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767461    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767466    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767470    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767475    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767479    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767484    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767488    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767493    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767496    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767501    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767505    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767509    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:35.774216 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767514    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767518    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767522    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767526    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767531    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767535    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767540    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767544    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767548    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767552    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767556    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767561    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767565    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767569    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767573    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767578    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767582    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767586    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767590    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:35.774580 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767597    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767602    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767607    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767612    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767617    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767621    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767626    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767630    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767634    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767639    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767643    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767663    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767668    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767672    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767676    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767680    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767684    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767688    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767692    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767696    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:35.775054 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767701    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767705    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767709    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767713    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767717    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767721    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767725    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767729    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767733    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767738    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767742    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767746    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767751    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767755    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767759    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767763    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767767    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767771    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767778    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:35.775538 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767783    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767787    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767792    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767796    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767800    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767804    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767808    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767813    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767817    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767822    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767826    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767830    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767834    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767838    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767842    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767847    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767851    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767856    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767862    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:35.776007 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767867    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767872    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767877    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767882    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767887    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767892    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767896    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767901    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:35.767906    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.767913    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.768058    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.771400    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.772441    2578 server.go:1019] "Starting client certificate rotation"
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.772534    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:15:35.776466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.772565    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:15:35.776855 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.775106    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:15:35.776855 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.776271    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:15:35.787613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.787594    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 11 18:15:35.790602 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.790585    2578 log.go:25] "Validated CRI v1 image API"
Sep 11 18:15:35.791483 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.791463    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 18:15:35.795701 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.795683    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b44419a7-0987-48a7-9692-49a88050048a:/dev/nvme0n1p3 f3b23e8c-8a1b-4f61-bd35-e3d722b300f4:/dev/nvme0n1p4]
Sep 11 18:15:35.795767 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.795701    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 18:15:35.801584 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.800935    2578 manager.go:216] Machine: {Timestamp:2026-09-11 18:15:35.79945736 +0000 UTC m=+0.378721913 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088989 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d1c02e0c0d4064a38c9f0e5f8047e SystemUUID:ec2d1c02-e0c0-d406-4a38-c9f0e5f8047e BootID:8d91ea66-1e01-415f-a563-99baf9885ef4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a7:36:d9:3f:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a7:36:d9:3f:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:d3:33:63:67:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 18:15:35.801584 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.801269    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 18:15:35.801584 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.801391    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 18:15:35.803481 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.803436    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 18:15:35.803695 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.803484    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-186.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 18:15:35.804265 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.804252    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 18:15:35.804313 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.804268    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 18:15:35.804313 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.804276    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:15:35.804861 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.804851    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:15:35.805654 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.805634    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 18:15:35.805775 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.805755    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:15:35.809334 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.809321    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:15:35.809446 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.809437    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 18:15:35.809486 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.809466    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 18:15:35.809486 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.809476    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 18:15:35.809486 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.809487    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 18:15:35.809610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.809495    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 11 18:15:35.809610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.809504    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 18:15:35.809869 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.809848    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:15:35.809964 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.809873    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:15:35.813446 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.813428    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 18:15:35.815557 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.815538    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 18:15:35.815668 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.815579    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 18:15:35.817756 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817617    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 18:15:35.817839 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817765    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 18:15:35.817839 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817776    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 18:15:35.817839 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817785    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 18:15:35.817839 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817798    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 18:15:35.817839 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817807    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 18:15:35.817839 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817828    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 18:15:35.818063 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817848    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 18:15:35.818063 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817860    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 18:15:35.818063 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817869    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 18:15:35.818063 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817890    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 18:15:35.818063 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.817899    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 18:15:35.818486 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.818474    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 18:15:35.818543 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.818488    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 18:15:35.819562 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.819542    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-186.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 18:15:35.819623 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.819543    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 18:15:35.821506 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.821490    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-186.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 18:15:35.822103 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.822091    2578 server.go:1268] "Started kubelet"
Sep 11 18:15:35.822185 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.822160    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 18:15:35.822221 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.822158    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 18:15:35.822253 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.822219    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 18:15:35.822995 ip-10-0-139-186 systemd[1]: Started Kubernetes Kubelet.
Sep 11 18:15:35.823593 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.823562    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 18:15:35.826312 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.826296    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 18:15:35.828245 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.828219    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 18:15:35.830776 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.830760    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 18:15:35.830854 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.830780    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 18:15:35.830912 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.830859    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:35.831488 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.829871    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-186.ec2.internal.18d456eed95fb638  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-186.ec2.internal,UID:ip-10-0-139-186.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-186.ec2.internal,},FirstTimestamp:2026-09-11 18:15:35.82206316 +0000 UTC m=+0.401327712,LastTimestamp:2026-09-11 18:15:35.82206316 +0000 UTC m=+0.401327712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-186.ec2.internal,}"
Sep 11 18:15:35.831488 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.831328    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 18:15:35.831488 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.831328    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 18:15:35.831488 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.831360    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 18:15:35.831488 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.831472    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 18:15:35.831488 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.831482    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 18:15:35.832380 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.832352    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 18:15:35.832380 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.832367    2578 factory.go:55] Registering systemd factory
Sep 11 18:15:35.832380 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.832373    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 11 18:15:35.833337 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.833325    2578 factory.go:153] Registering CRI-O factory
Sep 11 18:15:35.833387 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.833342    2578 factory.go:223] Registration of the crio container factory successfully
Sep 11 18:15:35.833387 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.833375    2578 factory.go:103] Registering Raw factory
Sep 11 18:15:35.833472 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.833389    2578 manager.go:1196] Started watching for new ooms in manager
Sep 11 18:15:35.833958 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.833944    2578 manager.go:349] Starting recovery of all containers
Sep 11 18:15:35.839568 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.839540    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-186.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 18:15:35.839699 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.839660    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 18:15:35.847979 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.847961    2578 manager.go:354] Recovery completed
Sep 11 18:15:35.853364 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.853350    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:35.855996 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.855969    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:35.856061 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.856013    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:35.856061 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.856026    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:35.856627 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.856614    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 18:15:35.856627 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.856626    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 18:15:35.856780 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.856615    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:35.856780 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.856641    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:15:35.857416 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.857393    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:35.857416 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.857418    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:35.857494 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.857428    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:35.858383 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.858312    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-186.ec2.internal.18d456eedb657f52  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-186.ec2.internal,UID:ip-10-0-139-186.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-186.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-186.ec2.internal,},FirstTimestamp:2026-09-11 18:15:35.855996754 +0000 UTC m=+0.435261312,LastTimestamp:2026-09-11 18:15:35.855996754 +0000 UTC m=+0.435261312,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-186.ec2.internal,}"
Sep 11 18:15:35.858686 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.858669    2578 policy_none.go:49] "None policy: Start"
Sep 11 18:15:35.858686 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.858686    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 18:15:35.858819 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.858696    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 18:15:35.860523 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.860512    2578 policy_none.go:47] "Start"
Sep 11 18:15:35.860834 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.860818    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r87mc"
Sep 11 18:15:35.869015 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.868987    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r87mc"
Sep 11 18:15:35.869438 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.869360    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-186.ec2.internal.18d456eedb65db9e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-186.ec2.internal,UID:ip-10-0-139-186.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-186.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-186.ec2.internal,},FirstTimestamp:2026-09-11 18:15:35.856020382 +0000 UTC m=+0.435284939,LastTimestamp:2026-09-11 18:15:35.856020382 +0000 UTC m=+0.435284939,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-186.ec2.internal,}"
Sep 11 18:15:35.916248 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.892276    2578 manager.go:337] "Starting Device Plugin manager"
Sep 11 18:15:35.916248 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.892322    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 18:15:35.916248 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.892358    2578 server.go:87] "Starting device plugin registration server"
Sep 11 18:15:35.916248 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.892720    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 18:15:35.916248 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.892731    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 18:15:35.916248 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.892830    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 18:15:35.916248 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.892935    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 18:15:35.916248 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.892945    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 18:15:35.916248 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.893469    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 18:15:35.916248 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.893518    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:35.929834 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.929802    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 18:15:35.930993 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.930976    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 18:15:35.931091 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.931021    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 18:15:35.931091 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.931044    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 18:15:35.931091 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:35.931081    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 18:15:35.934392 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.934374    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:35.993254 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.993225    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:35.994258 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.994236    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:35.994364 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.994272    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:35.994364 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.994282    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:35.994364 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:35.994305    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.002492 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.002436    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.002492 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.002456    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-186.ec2.internal\": node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:36.019876 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.019857    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:36.031423 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.031406    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal"]
Sep 11 18:15:36.031495 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.031461    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:36.032971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.032955    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:36.033050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.032980    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:36.033050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.032991    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:36.035007 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.034995    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:36.035157 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.035141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.035212 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.035173    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:36.035634 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.035610    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:36.035743 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.035640    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:36.035743 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.035671    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:36.035743 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.035724    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:36.035743 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.035744    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:36.035883 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.035756    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:36.037741 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.037725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.037804 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.037755    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:36.038408 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.038393    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:36.038487 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.038416    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:36.038487 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.038428    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:36.059234 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.059212    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-186.ec2.internal\" not found" node="ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.063829 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.063812    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-186.ec2.internal\" not found" node="ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.120485 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.120452    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:36.220948 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.220918    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:36.232149 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.232133    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be5ff56bb7e6310337c8daa0033c5d78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal\" (UID: \"be5ff56bb7e6310337c8daa0033c5d78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.232247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.232155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14d74969f57bbabba180bdcaf49f1bc5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-186.ec2.internal\" (UID: \"14d74969f57bbabba180bdcaf49f1bc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.232247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.232175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be5ff56bb7e6310337c8daa0033c5d78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal\" (UID: \"be5ff56bb7e6310337c8daa0033c5d78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.321102 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.321069    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:36.332959 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.332935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be5ff56bb7e6310337c8daa0033c5d78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal\" (UID: \"be5ff56bb7e6310337c8daa0033c5d78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.333078 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.332974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be5ff56bb7e6310337c8daa0033c5d78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal\" (UID: \"be5ff56bb7e6310337c8daa0033c5d78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.333078 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.333002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14d74969f57bbabba180bdcaf49f1bc5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-186.ec2.internal\" (UID: \"14d74969f57bbabba180bdcaf49f1bc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.333078 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.333045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be5ff56bb7e6310337c8daa0033c5d78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal\" (UID: \"be5ff56bb7e6310337c8daa0033c5d78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.333078 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.333059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be5ff56bb7e6310337c8daa0033c5d78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal\" (UID: \"be5ff56bb7e6310337c8daa0033c5d78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.333230 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.333042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14d74969f57bbabba180bdcaf49f1bc5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-186.ec2.internal\" (UID: \"14d74969f57bbabba180bdcaf49f1bc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.362614 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.362587    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.366683 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.366648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.421454 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.421428    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:36.528019 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.527960    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:36.628567 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.628492    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:36.729246 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.729206    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:36.772234 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.772203    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 18:15:36.772767 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.772420    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 18:15:36.829866 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:36.829846    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-186.ec2.internal\" not found"
Sep 11 18:15:36.874204 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.874162    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 18:10:36 +0000 UTC" deadline="2028-04-01 17:26:36.856867425 +0000 UTC"
Sep 11 18:15:36.874204 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.874201    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13631h10m59.98267238s"
Sep 11 18:15:36.896067 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.896009    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:36.931126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.931097    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.943960 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.943939    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:15:36.946152 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.946135    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal"
Sep 11 18:15:36.949765 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:36.949736    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d74969f57bbabba180bdcaf49f1bc5.slice/crio-e02030440d156ffae0091b1f99b04d36003ace0e6a869f482204a3712b2add46 WatchSource:0}: Error finding container e02030440d156ffae0091b1f99b04d36003ace0e6a869f482204a3712b2add46: Status 404 returned error can't find the container with id e02030440d156ffae0091b1f99b04d36003ace0e6a869f482204a3712b2add46
Sep 11 18:15:36.956318 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.956300    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:15:36.962755 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:36.962737    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:15:36.976132 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:36.976108    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5ff56bb7e6310337c8daa0033c5d78.slice/crio-ca23538bc463945a16d776249ef34fe8bfcbcd2f3aa2cebd1655f05ce38d000f WatchSource:0}: Error finding container ca23538bc463945a16d776249ef34fe8bfcbcd2f3aa2cebd1655f05ce38d000f: Status 404 returned error can't find the container with id ca23538bc463945a16d776249ef34fe8bfcbcd2f3aa2cebd1655f05ce38d000f
Sep 11 18:15:37.342331 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.342244    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:37.386352 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.386319    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:37.729193 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.729097    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:37.811865 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.811820    2578 apiserver.go:52] "Watching apiserver"
Sep 11 18:15:37.818622 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.818590    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 18:15:37.819046 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.819018    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-ztzmx","openshift-cluster-node-tuning-operator/tuned-z48mp","openshift-dns/node-resolver-2x9c6","openshift-multus/multus-additional-cni-plugins-hhrj4","openshift-multus/multus-p27kp","openshift-network-operator/iptables-alerter-qj8sr","openshift-ovn-kubernetes/ovnkube-node-pfrlp","kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2","openshift-image-registry/node-ca-l4h5q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal","openshift-multus/network-metrics-daemon-sv62g","openshift-network-diagnostics/network-check-target-g6jp5","kube-system/global-pull-secret-syncer-65zrr"]
Sep 11 18:15:37.822001 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.821978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.825013 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.824974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:37.825140 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:37.825082    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:37.825213 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.825199    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 18:15:37.825269 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.825213    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 18:15:37.825493 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.825470    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 18:15:37.825657 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.825498    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 18:15:37.825657 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.825620    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 18:15:37.825811 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.825787    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2s75l\""
Sep 11 18:15:37.825933 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.825900    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 18:15:37.827443 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.827402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:37.829782 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.829762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.830238 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.830198    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 18:15:37.830238 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.830229    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 18:15:37.830422 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.830405    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d4pvq\""
Sep 11 18:15:37.831647 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.831116    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 18:15:37.832304 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.832283    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.832711 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.832575    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 18:15:37.832711 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.832598    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 18:15:37.832711 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.832703    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4f2fg\""
Sep 11 18:15:37.833022 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.832999    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 18:15:37.833751 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.833729    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 18:15:37.834947 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.834639    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 18:15:37.834947 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.834727    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:37.835702 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.835635    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gkf6t\""
Sep 11 18:15:37.838296 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.836389    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 18:15:37.838296 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.838173    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:37.840186 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.840131    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:37.840387 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.840354    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 18:15:37.840573 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.840550    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2vckq\""
Sep 11 18:15:37.840739 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.840697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-var-lib-openvswitch\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.840828 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.840738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:37.840828 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.840764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-run-openvswitch\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.840828 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.840800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-ovnkube-script-lib\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.840979 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.840831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rtp\" (UniqueName: \"kubernetes.io/projected/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-kube-api-access-j6rtp\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.840979 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.840924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2c0153c-5b15-490d-93cc-d06a4fd9c434-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.840979 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.840969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2c0153c-5b15-490d-93cc-d06a4fd9c434-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.841134 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-run-systemd\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.841134 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-node-log\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.841134 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.841134 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-env-overrides\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.841339 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gp5\" (UniqueName: \"kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5\") pod \"network-check-target-g6jp5\" (UID: \"b0124408-a051-41e7-916e-ef1d331fc181\") " pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:37.841339 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-run-k8s-cni-cncf-io\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.841339 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-run-netns\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.841339 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-var-lib-cni-multus\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.841552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-kubelet\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.841552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-etc-openvswitch\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.841655 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2c0153c-5b15-490d-93cc-d06a4fd9c434-cnibin\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.841655 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2c0153c-5b15-490d-93cc-d06a4fd9c434-cni-binary-copy\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.841789 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.841670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-system-cni-dir\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842109 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-socket-dir-parent\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842232 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-hostroot\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842232 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-conf-dir\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842232 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-slash\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.842232 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8sz\" (UniqueName: \"kubernetes.io/projected/a2c0153c-5b15-490d-93cc-d06a4fd9c434-kube-api-access-4g8sz\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.842232 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-daemon-config\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-run-multus-certs\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-etc-kubernetes\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-ovn-node-metrics-cert\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.842479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/101bbe73-40f7-41b8-89a2-b09ba7ef75ca-hosts-file\") pod \"node-resolver-2x9c6\" (UID: \"101bbe73-40f7-41b8-89a2-b09ba7ef75ca\") " pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:37.842479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmxtv\" (UniqueName: \"kubernetes.io/projected/101bbe73-40f7-41b8-89a2-b09ba7ef75ca-kube-api-access-lmxtv\") pod \"node-resolver-2x9c6\" (UID: \"101bbe73-40f7-41b8-89a2-b09ba7ef75ca\") " pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:37.842479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-cni-dir\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-run-ovn\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.842479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.842479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-ovnkube-config\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.842479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/101bbe73-40f7-41b8-89a2-b09ba7ef75ca-tmp-dir\") pod \"node-resolver-2x9c6\" (UID: \"101bbe73-40f7-41b8-89a2-b09ba7ef75ca\") " pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-cni-binary-copy\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-var-lib-kubelet\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw959\" (UniqueName: \"kubernetes.io/projected/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-kube-api-access-dw959\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-cnibin\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-var-lib-cni-bin\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-systemd-units\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-run-netns\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-log-socket\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-cni-bin\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-cni-netd\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2c0153c-5b15-490d-93cc-d06a4fd9c434-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.842971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.842882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2c0153c-5b15-490d-93cc-d06a4fd9c434-system-cni-dir\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.844024 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.844003    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:37.844140 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.844122    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 18:15:37.844220 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.844184    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 18:15:37.844454 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.844434    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 18:15:37.844548 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.844508    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:15:37.844633 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.844615    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lpfpx\""
Sep 11 18:15:37.846166 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.846145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.847855 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.847826    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g5lx4\""
Sep 11 18:15:37.847984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.847971    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 18:15:37.848066 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.848049    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 18:15:37.848795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.848771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:37.851521 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.850783    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gs4jh\""
Sep 11 18:15:37.851521 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.851083    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:37.851521 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.851314    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:37.851697 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.851569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:37.851697 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:37.851618    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:37.853771 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.853748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:37.853872 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:37.853835    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:37.854779 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.854755    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 18:15:37.855015 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.854987    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 18:15:37.855229 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.855215    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h78k6\""
Sep 11 18:15:37.855285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.855264    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 18:15:37.891695 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.891666    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bpqc9"
Sep 11 18:15:37.899876 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.899838    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bpqc9"
Sep 11 18:15:37.933471 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.933434    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 18:15:37.936699 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.936622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal" event={"ID":"be5ff56bb7e6310337c8daa0033c5d78","Type":"ContainerStarted","Data":"ca23538bc463945a16d776249ef34fe8bfcbcd2f3aa2cebd1655f05ce38d000f"}
Sep 11 18:15:37.938085 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.938058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal" event={"ID":"14d74969f57bbabba180bdcaf49f1bc5","Type":"ContainerStarted","Data":"e02030440d156ffae0091b1f99b04d36003ace0e6a869f482204a3712b2add46"}
Sep 11 18:15:37.943757 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.943729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-sys-fs\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:37.943883 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.943767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-host\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.943883 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.943802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-systemd-units\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.943883 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.943828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-log-socket\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.943883 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.943880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-log-socket\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.944084 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.943896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-cni-netd\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.944084 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.943955    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-systemd-units\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.944084 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.943975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2c0153c-5b15-490d-93cc-d06a4fd9c434-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.944084 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-cni-netd\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.944084 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2c0153c-5b15-490d-93cc-d06a4fd9c434-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.944288 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-conf-dir\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.944288 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:37.944288 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71216fc7-e4e7-4e9f-809e-434f11255b53-host-slash\") pod \"iptables-alerter-qj8sr\" (UID: \"71216fc7-e4e7-4e9f-809e-434f11255b53\") " pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:37.944452 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-var-lib-openvswitch\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.944452 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-run-openvswitch\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.944452 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rtp\" (UniqueName: \"kubernetes.io/projected/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-kube-api-access-j6rtp\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.944452 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-run-k8s-cni-cncf-io\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.944452 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/861c7885-6882-4d54-a19a-99b7c86dc58e-serviceca\") pod \"node-ca-l4h5q\" (UID: \"861c7885-6882-4d54-a19a-99b7c86dc58e\") " pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:37.944452 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:37.944452 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.944752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-env-overrides\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.944752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-lib-modules\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.944752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ff28623-8744-4a92-856d-060583d70a8f-tmp\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.944752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2c0153c-5b15-490d-93cc-d06a4fd9c434-cnibin\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.944752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-socket-dir-parent\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.944752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-device-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:37.944752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46wzw\" (UniqueName: \"kubernetes.io/projected/a786fa1e-4608-4450-9305-341667b2bb57-kube-api-access-46wzw\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:37.944752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-modprobe-d\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.944752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-var-lib-kubelet\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.944752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/861c7885-6882-4d54-a19a-99b7c86dc58e-host\") pod \"node-ca-l4h5q\" (UID: \"861c7885-6882-4d54-a19a-99b7c86dc58e\") " pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2c0153c-5b15-490d-93cc-d06a4fd9c434-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-systemd\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/101bbe73-40f7-41b8-89a2-b09ba7ef75ca-tmp-dir\") pod \"node-resolver-2x9c6\" (UID: \"101bbe73-40f7-41b8-89a2-b09ba7ef75ca\") " pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-conf-dir\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2c0153c-5b15-490d-93cc-d06a4fd9c434-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-socket-dir-parent\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-run-openvswitch\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-var-lib-openvswitch\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2c0153c-5b15-490d-93cc-d06a4fd9c434-cnibin\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-var-lib-kubelet\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.944960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-run-k8s-cni-cncf-io\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw959\" (UniqueName: \"kubernetes.io/projected/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-kube-api-access-dw959\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-var-lib-kubelet\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9129c4f7-d49b-471e-abe5-81dbfdce82eb-agent-certs\") pod \"konnectivity-agent-ztzmx\" (UID: \"9129c4f7-d49b-471e-abe5-81dbfdce82eb\") " pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:37.945247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvx94\" (UniqueName: \"kubernetes.io/projected/4ff28623-8744-4a92-856d-060583d70a8f-kube-api-access-tvx94\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-run-netns\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/101bbe73-40f7-41b8-89a2-b09ba7ef75ca-tmp-dir\") pod \"node-resolver-2x9c6\" (UID: \"101bbe73-40f7-41b8-89a2-b09ba7ef75ca\") " pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-cni-bin\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-run-netns\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-cni-bin\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-system-cni-dir\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2050106a-dd99-409d-9860-6887d827d3af-kubelet-config\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2c0153c-5b15-490d-93cc-d06a4fd9c434-system-cni-dir\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-ovnkube-script-lib\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.945902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-env-overrides\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-system-cni-dir\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.946072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2c0153c-5b15-490d-93cc-d06a4fd9c434-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2c0153c-5b15-490d-93cc-d06a4fd9c434-system-cni-dir\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-run-multus-certs\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-ovnkube-script-lib\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-etc-kubernetes\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-etc-kubernetes\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-socket-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-sysconfig\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-run-multus-certs\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-kubernetes\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2c0153c-5b15-490d-93cc-d06a4fd9c434-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-run-systemd\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-node-log\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.946742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gp5\" (UniqueName: \"kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5\") pod \"network-check-target-g6jp5\" (UID: \"b0124408-a051-41e7-916e-ef1d331fc181\") " pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:37.947379 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-run-netns\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.947379 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-node-log\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.947379 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-run-systemd\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.947379 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.946813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-var-lib-cni-multus\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.947379 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-run-netns\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.947379 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-var-lib-cni-multus\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.947379 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzzr\" (UniqueName: \"kubernetes.io/projected/71216fc7-e4e7-4e9f-809e-434f11255b53-kube-api-access-7xzzr\") pod \"iptables-alerter-qj8sr\" (UID: \"71216fc7-e4e7-4e9f-809e-434f11255b53\") " pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:37.947379 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-registration-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-kubelet\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-etc-openvswitch\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2c0153c-5b15-490d-93cc-d06a4fd9c434-cni-binary-copy\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-hostroot\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxz8\" (UniqueName: \"kubernetes.io/projected/861c7885-6882-4d54-a19a-99b7c86dc58e-kube-api-access-hwxz8\") pod \"node-ca-l4h5q\" (UID: \"861c7885-6882-4d54-a19a-99b7c86dc58e\") " pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqz89\" (UniqueName: \"kubernetes.io/projected/78c498ff-9632-4d5a-8dad-d1d9a94137e1-kube-api-access-lqz89\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9129c4f7-d49b-471e-abe5-81dbfdce82eb-konnectivity-ca\") pod \"konnectivity-agent-ztzmx\" (UID: \"9129c4f7-d49b-471e-abe5-81dbfdce82eb\") " pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-sysctl-conf\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-slash\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8sz\" (UniqueName: \"kubernetes.io/projected/a2c0153c-5b15-490d-93cc-d06a4fd9c434-kube-api-access-4g8sz\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-daemon-config\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.947768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71216fc7-e4e7-4e9f-809e-434f11255b53-iptables-alerter-script\") pod \"iptables-alerter-qj8sr\" (UID: \"71216fc7-e4e7-4e9f-809e-434f11255b53\") " pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-sysctl-d\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-run\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4ff28623-8744-4a92-856d-060583d70a8f-etc-tuned\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-ovn-node-metrics-cert\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/101bbe73-40f7-41b8-89a2-b09ba7ef75ca-hosts-file\") pod \"node-resolver-2x9c6\" (UID: \"101bbe73-40f7-41b8-89a2-b09ba7ef75ca\") " pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmxtv\" (UniqueName: \"kubernetes.io/projected/101bbe73-40f7-41b8-89a2-b09ba7ef75ca-kube-api-access-lmxtv\") pod \"node-resolver-2x9c6\" (UID: \"101bbe73-40f7-41b8-89a2-b09ba7ef75ca\") " pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-cni-dir\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.947994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-sys\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-run-ovn\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-ovnkube-config\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-cni-binary-copy\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-cnibin\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-var-lib-cni-bin\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2050106a-dd99-409d-9860-6887d827d3af-dbus\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-etc-selinux\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:37.948385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-kubelet\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.949207 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-etc-openvswitch\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.949207 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.948943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2c0153c-5b15-490d-93cc-d06a4fd9c434-cni-binary-copy\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.949207 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.949020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-hostroot\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.949207 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.949129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-slash\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.949859 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.949840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-run-ovn\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.950003 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.949926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-cnibin\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.950003 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.949939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.950003 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.949965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-daemon-config\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.950229 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.950063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-host-var-lib-cni-bin\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.950229 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.950085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-multus-cni-dir\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.950229 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.950134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/101bbe73-40f7-41b8-89a2-b09ba7ef75ca-hosts-file\") pod \"node-resolver-2x9c6\" (UID: \"101bbe73-40f7-41b8-89a2-b09ba7ef75ca\") " pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:37.950229 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.950175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-ovnkube-config\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.950519 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.950477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-cni-binary-copy\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.951542 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.951332    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 18:15:37.956022 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.955998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-ovn-node-metrics-cert\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.956585 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.956561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw959\" (UniqueName: \"kubernetes.io/projected/e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981-kube-api-access-dw959\") pod \"multus-p27kp\" (UID: \"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981\") " pod="openshift-multus/multus-p27kp"
Sep 11 18:15:37.956688 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.956562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rtp\" (UniqueName: \"kubernetes.io/projected/84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e-kube-api-access-j6rtp\") pod \"ovnkube-node-pfrlp\" (UID: \"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:37.961745 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:37.961720    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:37.961745 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:37.961746    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:37.961895 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:37.961762    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f6gp5 for pod openshift-network-diagnostics/network-check-target-g6jp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:37.961895 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:37.961829    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5 podName:b0124408-a051-41e7-916e-ef1d331fc181 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:38.461808977 +0000 UTC m=+3.041073518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f6gp5" (UniqueName: "kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5") pod "network-check-target-g6jp5" (UID: "b0124408-a051-41e7-916e-ef1d331fc181") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:37.973091 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.973058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8sz\" (UniqueName: \"kubernetes.io/projected/a2c0153c-5b15-490d-93cc-d06a4fd9c434-kube-api-access-4g8sz\") pod \"multus-additional-cni-plugins-hhrj4\" (UID: \"a2c0153c-5b15-490d-93cc-d06a4fd9c434\") " pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:37.973233 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:37.973128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmxtv\" (UniqueName: \"kubernetes.io/projected/101bbe73-40f7-41b8-89a2-b09ba7ef75ca-kube-api-access-lmxtv\") pod \"node-resolver-2x9c6\" (UID: \"101bbe73-40f7-41b8-89a2-b09ba7ef75ca\") " pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:38.049509 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2050106a-dd99-409d-9860-6887d827d3af-dbus\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:38.049509 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-etc-selinux\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.049509 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-sys-fs\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.049795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-etc-selinux\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.049795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049663    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-sys-fs\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.049795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-host\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.049795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-host\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.049795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71216fc7-e4e7-4e9f-809e-434f11255b53-host-slash\") pod \"iptables-alerter-qj8sr\" (UID: \"71216fc7-e4e7-4e9f-809e-434f11255b53\") " pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/861c7885-6882-4d54-a19a-99b7c86dc58e-serviceca\") pod \"node-ca-l4h5q\" (UID: \"861c7885-6882-4d54-a19a-99b7c86dc58e\") " pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71216fc7-e4e7-4e9f-809e-434f11255b53-host-slash\") pod \"iptables-alerter-qj8sr\" (UID: \"71216fc7-e4e7-4e9f-809e-434f11255b53\") " pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-lib-modules\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ff28623-8744-4a92-856d-060583d70a8f-tmp\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2050106a-dd99-409d-9860-6887d827d3af-dbus\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.049922    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.049938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-device-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.050014    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:38.050037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-lib-modules\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46wzw\" (UniqueName: \"kubernetes.io/projected/a786fa1e-4608-4450-9305-341667b2bb57-kube-api-access-46wzw\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-modprobe-d\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-device-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.050119    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs podName:78c498ff-9632-4d5a-8dad-d1d9a94137e1 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:38.550096942 +0000 UTC m=+3.129361502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs") pod "network-metrics-daemon-sv62g" (UID: "78c498ff-9632-4d5a-8dad-d1d9a94137e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-modprobe-d\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.050237    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret podName:2050106a-dd99-409d-9860-6887d827d3af nodeName:}" failed. No retries permitted until 2026-09-11 18:15:38.550222779 +0000 UTC m=+3.129487337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret") pod "global-pull-secret-syncer-65zrr" (UID: "2050106a-dd99-409d-9860-6887d827d3af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-var-lib-kubelet\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/861c7885-6882-4d54-a19a-99b7c86dc58e-serviceca\") pod \"node-ca-l4h5q\" (UID: \"861c7885-6882-4d54-a19a-99b7c86dc58e\") " pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-var-lib-kubelet\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/861c7885-6882-4d54-a19a-99b7c86dc58e-host\") pod \"node-ca-l4h5q\" (UID: \"861c7885-6882-4d54-a19a-99b7c86dc58e\") " pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-systemd\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9129c4f7-d49b-471e-abe5-81dbfdce82eb-agent-certs\") pod \"konnectivity-agent-ztzmx\" (UID: \"9129c4f7-d49b-471e-abe5-81dbfdce82eb\") " pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050526    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/861c7885-6882-4d54-a19a-99b7c86dc58e-host\") pod \"node-ca-l4h5q\" (UID: \"861c7885-6882-4d54-a19a-99b7c86dc58e\") " pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:38.050552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvx94\" (UniqueName: \"kubernetes.io/projected/4ff28623-8744-4a92-856d-060583d70a8f-kube-api-access-tvx94\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2050106a-dd99-409d-9860-6887d827d3af-kubelet-config\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-systemd\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-socket-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2050106a-dd99-409d-9860-6887d827d3af-kubelet-config\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-socket-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-sysconfig\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-kubernetes\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzzr\" (UniqueName: \"kubernetes.io/projected/71216fc7-e4e7-4e9f-809e-434f11255b53-kube-api-access-7xzzr\") pod \"iptables-alerter-qj8sr\" (UID: \"71216fc7-e4e7-4e9f-809e-434f11255b53\") " pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-registration-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-sysconfig\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxz8\" (UniqueName: \"kubernetes.io/projected/861c7885-6882-4d54-a19a-99b7c86dc58e-kube-api-access-hwxz8\") pod \"node-ca-l4h5q\" (UID: \"861c7885-6882-4d54-a19a-99b7c86dc58e\") " pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.050995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-kubernetes\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a786fa1e-4608-4450-9305-341667b2bb57-registration-dir\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqz89\" (UniqueName: \"kubernetes.io/projected/78c498ff-9632-4d5a-8dad-d1d9a94137e1-kube-api-access-lqz89\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:38.051268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9129c4f7-d49b-471e-abe5-81dbfdce82eb-konnectivity-ca\") pod \"konnectivity-agent-ztzmx\" (UID: \"9129c4f7-d49b-471e-abe5-81dbfdce82eb\") " pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-sysctl-conf\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71216fc7-e4e7-4e9f-809e-434f11255b53-iptables-alerter-script\") pod \"iptables-alerter-qj8sr\" (UID: \"71216fc7-e4e7-4e9f-809e-434f11255b53\") " pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-sysctl-d\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-run\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4ff28623-8744-4a92-856d-060583d70a8f-etc-tuned\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-sys\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-sysctl-d\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-etc-sysctl-conf\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-run\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ff28623-8744-4a92-856d-060583d70a8f-sys\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/71216fc7-e4e7-4e9f-809e-434f11255b53-iptables-alerter-script\") pod \"iptables-alerter-qj8sr\" (UID: \"71216fc7-e4e7-4e9f-809e-434f11255b53\") " pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:38.052132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.051871    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9129c4f7-d49b-471e-abe5-81dbfdce82eb-konnectivity-ca\") pod \"konnectivity-agent-ztzmx\" (UID: \"9129c4f7-d49b-471e-abe5-81dbfdce82eb\") " pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:38.053402 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.053348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ff28623-8744-4a92-856d-060583d70a8f-tmp\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.053549 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.053519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9129c4f7-d49b-471e-abe5-81dbfdce82eb-agent-certs\") pod \"konnectivity-agent-ztzmx\" (UID: \"9129c4f7-d49b-471e-abe5-81dbfdce82eb\") " pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:38.053673 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.053592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4ff28623-8744-4a92-856d-060583d70a8f-etc-tuned\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.079847 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.079813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqz89\" (UniqueName: \"kubernetes.io/projected/78c498ff-9632-4d5a-8dad-d1d9a94137e1-kube-api-access-lqz89\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:38.080922 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.080879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvx94\" (UniqueName: \"kubernetes.io/projected/4ff28623-8744-4a92-856d-060583d70a8f-kube-api-access-tvx94\") pod \"tuned-z48mp\" (UID: \"4ff28623-8744-4a92-856d-060583d70a8f\") " pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.081195 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.081173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzzr\" (UniqueName: \"kubernetes.io/projected/71216fc7-e4e7-4e9f-809e-434f11255b53-kube-api-access-7xzzr\") pod \"iptables-alerter-qj8sr\" (UID: \"71216fc7-e4e7-4e9f-809e-434f11255b53\") " pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:38.081271 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.081173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46wzw\" (UniqueName: \"kubernetes.io/projected/a786fa1e-4608-4450-9305-341667b2bb57-kube-api-access-46wzw\") pod \"aws-ebs-csi-driver-node-f6hd2\" (UID: \"a786fa1e-4608-4450-9305-341667b2bb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.081657 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.081636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxz8\" (UniqueName: \"kubernetes.io/projected/861c7885-6882-4d54-a19a-99b7c86dc58e-kube-api-access-hwxz8\") pod \"node-ca-l4h5q\" (UID: \"861c7885-6882-4d54-a19a-99b7c86dc58e\") " pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:38.137705 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.137672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:15:38.145218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.145189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2x9c6"
Sep 11 18:15:38.155401 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.155375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hhrj4"
Sep 11 18:15:38.166427 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.166395    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p27kp"
Sep 11 18:15:38.173121 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.173101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qj8sr"
Sep 11 18:15:38.180996 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.180974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2"
Sep 11 18:15:38.189352 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.189336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:38.196054 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.196030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z48mp"
Sep 11 18:15:38.201907 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.201884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l4h5q"
Sep 11 18:15:38.556109 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.556065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:38.556109 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.556111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:38.556359 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.556200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gp5\" (UniqueName: \"kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5\") pod \"network-check-target-g6jp5\" (UID: \"b0124408-a051-41e7-916e-ef1d331fc181\") " pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:38.556359 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.556219    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:38.556359 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.556279    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs podName:78c498ff-9632-4d5a-8dad-d1d9a94137e1 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:39.55626383 +0000 UTC m=+4.135528389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs") pod "network-metrics-daemon-sv62g" (UID: "78c498ff-9632-4d5a-8dad-d1d9a94137e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:38.556359 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.556322    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:38.556569 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.556378    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret podName:2050106a-dd99-409d-9860-6887d827d3af nodeName:}" failed. No retries permitted until 2026-09-11 18:15:39.556365331 +0000 UTC m=+4.135629889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret") pod "global-pull-secret-syncer-65zrr" (UID: "2050106a-dd99-409d-9860-6887d827d3af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:38.556569 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.556325    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:38.556569 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.556413    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:38.556569 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.556425    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f6gp5 for pod openshift-network-diagnostics/network-check-target-g6jp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:38.556569 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:38.556471    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5 podName:b0124408-a051-41e7-916e-ef1d331fc181 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:39.556449494 +0000 UTC m=+4.135714043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6gp5" (UniqueName: "kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5") pod "network-check-target-g6jp5" (UID: "b0124408-a051-41e7-916e-ef1d331fc181") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:38.902907 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.902825    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:38 +0000 UTC" deadline="2028-04-17 16:15:26.268527865 +0000 UTC"
Sep 11 18:15:38.902907 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.902856    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14013h59m47.365675012s"
Sep 11 18:15:38.932179 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:38.932148    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101bbe73_40f7_41b8_89a2_b09ba7ef75ca.slice/crio-5334ab7b055025e2f62b648b225c658451160ef6b700da7ee91a90e42eacdc38 WatchSource:0}: Error finding container 5334ab7b055025e2f62b648b225c658451160ef6b700da7ee91a90e42eacdc38: Status 404 returned error can't find the container with id 5334ab7b055025e2f62b648b225c658451160ef6b700da7ee91a90e42eacdc38
Sep 11 18:15:38.933604 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:38.933559    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda786fa1e_4608_4450_9305_341667b2bb57.slice/crio-0ebf788de7721fbce0da3f34edd7b8c4af6e923e2fbe5beb3bfa98cb0490b142 WatchSource:0}: Error finding container 0ebf788de7721fbce0da3f34edd7b8c4af6e923e2fbe5beb3bfa98cb0490b142: Status 404 returned error can't find the container with id 0ebf788de7721fbce0da3f34edd7b8c4af6e923e2fbe5beb3bfa98cb0490b142
Sep 11 18:15:38.935382 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:38.934614    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bcfe1f_f59a_4f4a_9ce7_90f0280aac7e.slice/crio-5ccae90915e6613952ab16fb1ceba2433708e90cda4c2e68e02002f5c872cc52 WatchSource:0}: Error finding container 5ccae90915e6613952ab16fb1ceba2433708e90cda4c2e68e02002f5c872cc52: Status 404 returned error can't find the container with id 5ccae90915e6613952ab16fb1ceba2433708e90cda4c2e68e02002f5c872cc52
Sep 11 18:15:38.937300 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:38.937279    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c0153c_5b15_490d_93cc_d06a4fd9c434.slice/crio-61f1b07d4df71af17c50172fe92a799ac94f3887dce4cec7196dffaaf113c4a6 WatchSource:0}: Error finding container 61f1b07d4df71af17c50172fe92a799ac94f3887dce4cec7196dffaaf113c4a6: Status 404 returned error can't find the container with id 61f1b07d4df71af17c50172fe92a799ac94f3887dce4cec7196dffaaf113c4a6
Sep 11 18:15:38.939522 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:38.939487    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45aafb7_2c22_4dff_b4bd_ecf0a8ea2981.slice/crio-d1374d76ddfb196bac17252ff8f60c63f5498b357fd36720280900b22b9a4f56 WatchSource:0}: Error finding container d1374d76ddfb196bac17252ff8f60c63f5498b357fd36720280900b22b9a4f56: Status 404 returned error can't find the container with id d1374d76ddfb196bac17252ff8f60c63f5498b357fd36720280900b22b9a4f56
Sep 11 18:15:38.939857 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:38.939834    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861c7885_6882_4d54_a19a_99b7c86dc58e.slice/crio-3a4a3e424c748479c9756addc23f9314ac41d58a6a8099f362b32e9510d397c4 WatchSource:0}: Error finding container 3a4a3e424c748479c9756addc23f9314ac41d58a6a8099f362b32e9510d397c4: Status 404 returned error can't find the container with id 3a4a3e424c748479c9756addc23f9314ac41d58a6a8099f362b32e9510d397c4
Sep 11 18:15:38.941777 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:15:38.941761    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff28623_8744_4a92_856d_060583d70a8f.slice/crio-1e4ac45f034a2ae2c0de8cdfb25f898bcfbcfeb4626804c6c4dec6f2d8c8be0c WatchSource:0}: Error finding container 1e4ac45f034a2ae2c0de8cdfb25f898bcfbcfeb4626804c6c4dec6f2d8c8be0c: Status 404 returned error can't find the container with id 1e4ac45f034a2ae2c0de8cdfb25f898bcfbcfeb4626804c6c4dec6f2d8c8be0c
Sep 11 18:15:38.942674 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.942594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2x9c6" event={"ID":"101bbe73-40f7-41b8-89a2-b09ba7ef75ca","Type":"ContainerStarted","Data":"5334ab7b055025e2f62b648b225c658451160ef6b700da7ee91a90e42eacdc38"}
Sep 11 18:15:38.945301 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.945268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhrj4" event={"ID":"a2c0153c-5b15-490d-93cc-d06a4fd9c434","Type":"ContainerStarted","Data":"61f1b07d4df71af17c50172fe92a799ac94f3887dce4cec7196dffaaf113c4a6"}
Sep 11 18:15:38.948157 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.948130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp" event={"ID":"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e","Type":"ContainerStarted","Data":"5ccae90915e6613952ab16fb1ceba2433708e90cda4c2e68e02002f5c872cc52"}
Sep 11 18:15:38.949544 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:38.949515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2" event={"ID":"a786fa1e-4608-4450-9305-341667b2bb57","Type":"ContainerStarted","Data":"0ebf788de7721fbce0da3f34edd7b8c4af6e923e2fbe5beb3bfa98cb0490b142"}
Sep 11 18:15:39.565434 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.565339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gp5\" (UniqueName: \"kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5\") pod \"network-check-target-g6jp5\" (UID: \"b0124408-a051-41e7-916e-ef1d331fc181\") " pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:39.565434 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.565431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:39.565666 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.565464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:39.565666 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.565597    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:39.565800 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.565678    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs podName:78c498ff-9632-4d5a-8dad-d1d9a94137e1 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:41.565656101 +0000 UTC m=+6.144920659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs") pod "network-metrics-daemon-sv62g" (UID: "78c498ff-9632-4d5a-8dad-d1d9a94137e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:39.566178 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.566157    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:39.566246 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.566186    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:39.566246 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.566203    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f6gp5 for pod openshift-network-diagnostics/network-check-target-g6jp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:39.566356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.566252    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5 podName:b0124408-a051-41e7-916e-ef1d331fc181 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:41.566235586 +0000 UTC m=+6.145500143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6gp5" (UniqueName: "kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5") pod "network-check-target-g6jp5" (UID: "b0124408-a051-41e7-916e-ef1d331fc181") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:39.566356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.566314    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:39.566356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.566351    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret podName:2050106a-dd99-409d-9860-6887d827d3af nodeName:}" failed. No retries permitted until 2026-09-11 18:15:41.566338457 +0000 UTC m=+6.145603015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret") pod "global-pull-secret-syncer-65zrr" (UID: "2050106a-dd99-409d-9860-6887d827d3af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:39.903218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.903103    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:38 +0000 UTC" deadline="2028-05-12 16:59:37.003748295 +0000 UTC"
Sep 11 18:15:39.903218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.903165    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14614h43m57.100587369s"
Sep 11 18:15:39.935052 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.934194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:39.935052 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.934329    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:39.935052 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.934770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:39.935052 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.934872    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:39.935052 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.934933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:39.935052 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:39.934997    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:39.962817 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.962785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ztzmx" event={"ID":"9129c4f7-d49b-471e-abe5-81dbfdce82eb","Type":"ContainerStarted","Data":"bad9ec9c88b1489aa140f568897fdb3e7b7653263c747fa61346dfe2555eda69"}
Sep 11 18:15:39.974801 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.974772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l4h5q" event={"ID":"861c7885-6882-4d54-a19a-99b7c86dc58e","Type":"ContainerStarted","Data":"3a4a3e424c748479c9756addc23f9314ac41d58a6a8099f362b32e9510d397c4"}
Sep 11 18:15:39.983835 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.983811    2578 generic.go:356] "Generic (PLEG): container finished" podID="be5ff56bb7e6310337c8daa0033c5d78" containerID="4c6afe4cf67a9cebab3f35b62ff468f4c94fbc96503be3b02687506ebd3ad70d" exitCode=0
Sep 11 18:15:39.984452 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.984427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal" event={"ID":"be5ff56bb7e6310337c8daa0033c5d78","Type":"ContainerDied","Data":"4c6afe4cf67a9cebab3f35b62ff468f4c94fbc96503be3b02687506ebd3ad70d"}
Sep 11 18:15:39.990256 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.990227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal" event={"ID":"14d74969f57bbabba180bdcaf49f1bc5","Type":"ContainerStarted","Data":"97f880dd081aae1535da22fc6d6bf7d708087f81e723f23981d36e8f58bd0c05"}
Sep 11 18:15:39.999138 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:39.999109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qj8sr" event={"ID":"71216fc7-e4e7-4e9f-809e-434f11255b53","Type":"ContainerStarted","Data":"062b0979937fd7217738d91ca030e907d4b7d38113941dfa9b637ac48f0f513b"}
Sep 11 18:15:40.006060 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:40.005986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z48mp" event={"ID":"4ff28623-8744-4a92-856d-060583d70a8f","Type":"ContainerStarted","Data":"1e4ac45f034a2ae2c0de8cdfb25f898bcfbcfeb4626804c6c4dec6f2d8c8be0c"}
Sep 11 18:15:40.024287 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:40.024214    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p27kp" event={"ID":"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981","Type":"ContainerStarted","Data":"d1374d76ddfb196bac17252ff8f60c63f5498b357fd36720280900b22b9a4f56"}
Sep 11 18:15:41.046605 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:41.044302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal" event={"ID":"be5ff56bb7e6310337c8daa0033c5d78","Type":"ContainerStarted","Data":"1098d1406af7459a897941cf33fb45d0c4bd6ce5045dbd06157bed0c1bad5f87"}
Sep 11 18:15:41.062251 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:41.062193    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-186.ec2.internal" podStartSLOduration=4.06216992 podStartE2EDuration="4.06216992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:40.01481863 +0000 UTC m=+4.594083200" watchObservedRunningTime="2026-09-11 18:15:41.06216992 +0000 UTC m=+5.641434486"
Sep 11 18:15:41.583488 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:41.583220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:41.583698 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:41.583517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:41.583698 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:41.583579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gp5\" (UniqueName: \"kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5\") pod \"network-check-target-g6jp5\" (UID: \"b0124408-a051-41e7-916e-ef1d331fc181\") " pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:41.583882 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.583745    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:41.583882 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.583768    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:41.583882 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.583782    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f6gp5 for pod openshift-network-diagnostics/network-check-target-g6jp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:41.583882 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.583848    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5 podName:b0124408-a051-41e7-916e-ef1d331fc181 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:45.583825465 +0000 UTC m=+10.163090031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6gp5" (UniqueName: "kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5") pod "network-check-target-g6jp5" (UID: "b0124408-a051-41e7-916e-ef1d331fc181") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:41.584112 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.583934    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:41.584112 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.583972    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs podName:78c498ff-9632-4d5a-8dad-d1d9a94137e1 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:45.583959949 +0000 UTC m=+10.163224508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs") pod "network-metrics-daemon-sv62g" (UID: "78c498ff-9632-4d5a-8dad-d1d9a94137e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:41.584404 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.584382    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:41.584484 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.584442    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret podName:2050106a-dd99-409d-9860-6887d827d3af nodeName:}" failed. No retries permitted until 2026-09-11 18:15:45.584425425 +0000 UTC m=+10.163689972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret") pod "global-pull-secret-syncer-65zrr" (UID: "2050106a-dd99-409d-9860-6887d827d3af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:41.935356 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:41.935274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:41.935521 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.935452    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:41.936315 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:41.935846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:41.936315 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.936021    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:41.936315 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:41.936115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:41.936315 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:41.936237    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:43.932253 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:43.932218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:43.932717 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:43.932357    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:43.932875 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:43.932824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:43.933008 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:43.932926    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:43.933008 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:43.933002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:43.933151 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:43.933091    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:45.618885 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:45.618778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gp5\" (UniqueName: \"kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5\") pod \"network-check-target-g6jp5\" (UID: \"b0124408-a051-41e7-916e-ef1d331fc181\") " pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:45.618885 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:45.618858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:45.619385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:45.618892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:45.619385 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.618985    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:45.619385 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.619007    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:45.619385 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.619013    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:45.619385 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.619020    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f6gp5 for pod openshift-network-diagnostics/network-check-target-g6jp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:45.619385 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.619075    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs podName:78c498ff-9632-4d5a-8dad-d1d9a94137e1 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:53.619055493 +0000 UTC m=+18.198320055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs") pod "network-metrics-daemon-sv62g" (UID: "78c498ff-9632-4d5a-8dad-d1d9a94137e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:45.619385 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.619095    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5 podName:b0124408-a051-41e7-916e-ef1d331fc181 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:53.619084934 +0000 UTC m=+18.198349476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6gp5" (UniqueName: "kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5") pod "network-check-target-g6jp5" (UID: "b0124408-a051-41e7-916e-ef1d331fc181") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:45.619385 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.619116    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:45.619385 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.619155    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret podName:2050106a-dd99-409d-9860-6887d827d3af nodeName:}" failed. No retries permitted until 2026-09-11 18:15:53.619138264 +0000 UTC m=+18.198402807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret") pod "global-pull-secret-syncer-65zrr" (UID: "2050106a-dd99-409d-9860-6887d827d3af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:45.935574 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:45.935491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:45.935574 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:45.935527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:45.935814 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.935624    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:45.935814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:45.935671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:45.935814 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.935762    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:45.935966 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:45.935837    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:47.934693 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:47.934644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:47.935146 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:47.934773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:47.935146 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:47.934783    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:47.935146 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:47.934849    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:47.935146 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:47.934877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:47.935146 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:47.934983    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:49.931361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:49.931333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:49.931361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:49.931349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:49.931814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:49.931333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:49.931814 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:49.931438    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:49.931814 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:49.931510    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:49.931814 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:49.931565    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:51.932055 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:51.932025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:51.932466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:51.932069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:51.932466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:51.932027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:51.932466 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:51.932146    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:51.932466 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:51.932224    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:51.932466 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:51.932295    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:53.678432 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:53.678393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:53.678432 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:53.678435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:53.678873 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:53.678471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gp5\" (UniqueName: \"kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5\") pod \"network-check-target-g6jp5\" (UID: \"b0124408-a051-41e7-916e-ef1d331fc181\") " pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:53.678873 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.678535    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:53.678873 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.678576    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:53.678873 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.678584    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:53.678873 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.678588    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:53.678873 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.678634    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f6gp5 for pod openshift-network-diagnostics/network-check-target-g6jp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:53.678873 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.678594    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret podName:2050106a-dd99-409d-9860-6887d827d3af nodeName:}" failed. No retries permitted until 2026-09-11 18:16:09.678578887 +0000 UTC m=+34.257843425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret") pod "global-pull-secret-syncer-65zrr" (UID: "2050106a-dd99-409d-9860-6887d827d3af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:53.678873 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.678688    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs podName:78c498ff-9632-4d5a-8dad-d1d9a94137e1 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:09.678675734 +0000 UTC m=+34.257940273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs") pod "network-metrics-daemon-sv62g" (UID: "78c498ff-9632-4d5a-8dad-d1d9a94137e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:53.678873 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.678699    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5 podName:b0124408-a051-41e7-916e-ef1d331fc181 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:09.678693219 +0000 UTC m=+34.257957758 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6gp5" (UniqueName: "kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5") pod "network-check-target-g6jp5" (UID: "b0124408-a051-41e7-916e-ef1d331fc181") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:53.931378 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:53.931291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:53.931378 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:53.931313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:53.931567 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.931423    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:53.931567 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.931477    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:53.931567 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:53.931497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:53.931678 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:53.931569    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:55.932405 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:55.932372    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:55.932842 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:55.932462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:55.932842 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:55.932503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:55.932842 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:55.932514    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:55.932842 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:55.932573    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:55.932842 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:55.932693    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:57.931771 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:57.931519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:57.932661 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:57.931525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:57.932661 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:57.931576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:57.932661 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:57.931888    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:57.932661 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:57.931970    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:57.932661 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:57.932045    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:15:58.080175 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.079972    2578 generic.go:356] "Generic (PLEG): container finished" podID="a2c0153c-5b15-490d-93cc-d06a4fd9c434" containerID="8163be24a75a6675184e0501e2e0edfe48dd8fcc522ebb839b62bc2426bebb6b" exitCode=0
Sep 11 18:15:58.080318 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.080058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhrj4" event={"ID":"a2c0153c-5b15-490d-93cc-d06a4fd9c434","Type":"ContainerDied","Data":"8163be24a75a6675184e0501e2e0edfe48dd8fcc522ebb839b62bc2426bebb6b"}
Sep 11 18:15:58.082284 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.082255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2" event={"ID":"a786fa1e-4608-4450-9305-341667b2bb57","Type":"ContainerStarted","Data":"4497a9f96e1068952608ddb94335d04d1fb68d9db23b37500e9296b835ae4423"}
Sep 11 18:15:58.084091 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.084055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2x9c6" event={"ID":"101bbe73-40f7-41b8-89a2-b09ba7ef75ca","Type":"ContainerStarted","Data":"95685a0d705fa0cd281638577c4d87551151031a1765d76cfdbfd3e38d154349"}
Sep 11 18:15:58.086521 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.086504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z48mp" event={"ID":"4ff28623-8744-4a92-856d-060583d70a8f","Type":"ContainerStarted","Data":"a14b893d5add0285d1426baf8ebb1492887cb208f90b652444c608429b0042cf"}
Sep 11 18:15:58.088795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.088765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p27kp" event={"ID":"e45aafb7-2c22-4dff-b4bd-ecf0a8ea2981","Type":"ContainerStarted","Data":"5df1ea22dd39a0cebce4cb6ae2f3fb9367858c42224b70ad996af7961bff8a29"}
Sep 11 18:15:58.092382 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.092363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp" event={"ID":"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e","Type":"ContainerStarted","Data":"df649759b61c23fc5238ea1c8a5431b214c869ca679ab979e00d07572515ab82"}
Sep 11 18:15:58.092473 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.092428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp" event={"ID":"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e","Type":"ContainerStarted","Data":"59863d974b06214d1d638d3d4e6cba79885ca075dc45332fb42368d782a395bb"}
Sep 11 18:15:58.092535 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.092471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp" event={"ID":"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e","Type":"ContainerStarted","Data":"053629f38b4e014b8b6a135cc1b27ae05e7724fcf9e00c1b63c2ee3141ddb343"}
Sep 11 18:15:58.092535 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.092516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp" event={"ID":"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e","Type":"ContainerStarted","Data":"2bf189f65ce00b5b0eb0d45a1be65fb6d0ad5737634421460bd276798cdbf3d1"}
Sep 11 18:15:58.092608 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.092545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp" event={"ID":"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e","Type":"ContainerStarted","Data":"c12503e8f6a8f9d7c4de8f1e6e0def7034ed1b5844bac308b73d74bf3e60c792"}
Sep 11 18:15:58.092608 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.092574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp" event={"ID":"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e","Type":"ContainerStarted","Data":"e532368dda60eb102ff5b96312b58d5b0d2aba11f6f06f65e0ff482aec18ea46"}
Sep 11 18:15:58.094171 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.094144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ztzmx" event={"ID":"9129c4f7-d49b-471e-abe5-81dbfdce82eb","Type":"ContainerStarted","Data":"75c8c741b4ada955cc2ac99a551736870200fe3e82fb274c54eb3d5cc6eeba67"}
Sep 11 18:15:58.095890 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.095870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l4h5q" event={"ID":"861c7885-6882-4d54-a19a-99b7c86dc58e","Type":"ContainerStarted","Data":"0a585c29273b297567abf47fb765236832cc4ce928fe3e0e948450b5fbd80668"}
Sep 11 18:15:58.102143 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.102095    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-186.ec2.internal" podStartSLOduration=21.1020794 podStartE2EDuration="21.1020794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:41.062698784 +0000 UTC m=+5.641963351" watchObservedRunningTime="2026-09-11 18:15:58.1020794 +0000 UTC m=+22.681343962"
Sep 11 18:15:58.116565 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.116527    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2x9c6" podStartSLOduration=3.807897371 podStartE2EDuration="22.116515243s" totalImagesPullingTime="18.308617872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:58.11641635 +0000 UTC m=+22.695680915" watchObservedRunningTime="2026-09-11 18:15:58.116515243 +0000 UTC m=+22.695779803"
Sep 11 18:15:58.133344 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.133307    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p27kp" podStartSLOduration=3.775461216 podStartE2EDuration="22.133293913s" totalImagesPullingTime="18.357832697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:58.132952902 +0000 UTC m=+22.712217464" watchObservedRunningTime="2026-09-11 18:15:58.133293913 +0000 UTC m=+22.712558473"
Sep 11 18:15:58.170062 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.170022    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z48mp" podStartSLOduration=3.847524957 podStartE2EDuration="22.170009844s" totalImagesPullingTime="18.322484887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:58.169862976 +0000 UTC m=+22.749127538" watchObservedRunningTime="2026-09-11 18:15:58.170009844 +0000 UTC m=+22.749274402"
Sep 11 18:15:58.170729 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.170694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ztzmx" podStartSLOduration=7.4308542 podStartE2EDuration="22.170681711s" totalImagesPullingTime="14.739827511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:58.151924155 +0000 UTC m=+22.731188719" watchObservedRunningTime="2026-09-11 18:15:58.170681711 +0000 UTC m=+22.749946272"
Sep 11 18:15:58.188261 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.188218    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l4h5q" podStartSLOduration=3.886741189 podStartE2EDuration="22.188202536s" totalImagesPullingTime="18.301461347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:58.18773504 +0000 UTC m=+22.766999602" watchObservedRunningTime="2026-09-11 18:15:58.188202536 +0000 UTC m=+22.767467100"
Sep 11 18:15:58.581441 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.581417    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 18:15:58.774487 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.774414    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:58.775196 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.775173    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:58.908051 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.907947    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T18:15:58.581438157Z","UUID":"826fa1c0-bafc-44bf-be2e-f7fceb91ddb3","Handler":null,"Name":"","Endpoint":""}
Sep 11 18:15:58.909757 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.909733    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 18:15:58.909895 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:58.909765    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 18:15:59.100951 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:59.100860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2" event={"ID":"a786fa1e-4608-4450-9305-341667b2bb57","Type":"ContainerStarted","Data":"c3d2ae5a282df08a608179b3c846f5c7063183313e53729ebd97c11cee1cc86b"}
Sep 11 18:15:59.103602 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:59.103564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qj8sr" event={"ID":"71216fc7-e4e7-4e9f-809e-434f11255b53","Type":"ContainerStarted","Data":"a9ae4d2bd0c17a17b1bf82a5e9388deb064eaf642bab26e151b9d2c65eacbf16"}
Sep 11 18:15:59.104082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:59.104061    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:59.104921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:59.104904    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ztzmx"
Sep 11 18:15:59.121591 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:59.121541    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qj8sr" podStartSLOduration=4.826583031 podStartE2EDuration="23.121523604s" totalImagesPullingTime="18.294940573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:59.121211357 +0000 UTC m=+23.700475919" watchObservedRunningTime="2026-09-11 18:15:59.121523604 +0000 UTC m=+23.700788166"
Sep 11 18:15:59.931599 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:59.931522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:15:59.931599 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:59.931533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:15:59.931878 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:15:59.931643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:15:59.931878 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:59.931661    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:15:59.931878 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:59.931757    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:15:59.931878 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:15:59.931824    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:16:00.116822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:00.116787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2" event={"ID":"a786fa1e-4608-4450-9305-341667b2bb57","Type":"ContainerStarted","Data":"710222ac81829f6bde91c90338a665f5ec0f65db14c536c6c332f83025a578ab"}
Sep 11 18:16:00.138242 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:00.138199    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6hd2" podStartSLOduration=3.885329481 podStartE2EDuration="24.138182609s" totalImagesPullingTime="20.252853128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:00.137541995 +0000 UTC m=+24.716806547" watchObservedRunningTime="2026-09-11 18:16:00.138182609 +0000 UTC m=+24.717447167"
Sep 11 18:16:01.123529 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:01.123492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp" event={"ID":"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e","Type":"ContainerStarted","Data":"c9a77f94360763f97359923a59025bf71d99f8e99c8949cb5372c4893c2f95c5"}
Sep 11 18:16:01.931576 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:01.931536    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:01.931576 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:01.931574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:01.931838 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:01.931549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:16:01.931838 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:01.931688    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:16:01.931838 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:01.931773    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:16:01.931974 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:01.931849    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:16:03.131475 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:03.131240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp" event={"ID":"84bcfe1f-f59a-4f4a-9ce7-90f0280aac7e","Type":"ContainerStarted","Data":"076ea7960f858425298d0f1507d3176137c95fdc9b7c546f50aa8a69fa0652ba"}
Sep 11 18:16:03.132158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:03.131529    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:16:03.133154 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:03.133115    2578 generic.go:356] "Generic (PLEG): container finished" podID="a2c0153c-5b15-490d-93cc-d06a4fd9c434" containerID="a49df149108e690a4dfcfd4d4893fa974b2f614ba412603b7525a0327e125413" exitCode=0
Sep 11 18:16:03.133378 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:03.133159    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhrj4" event={"ID":"a2c0153c-5b15-490d-93cc-d06a4fd9c434","Type":"ContainerDied","Data":"a49df149108e690a4dfcfd4d4893fa974b2f614ba412603b7525a0327e125413"}
Sep 11 18:16:03.146782 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:03.146764    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:16:03.163438 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:03.163400    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp" podStartSLOduration=8.79118787 podStartE2EDuration="27.163386954s" totalImagesPullingTime="18.372199084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:03.161673055 +0000 UTC m=+27.740937608" watchObservedRunningTime="2026-09-11 18:16:03.163386954 +0000 UTC m=+27.742651515"
Sep 11 18:16:03.905358 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:03.905328    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:16:03.931493 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:03.931347    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:16:03.931593 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:03.931369    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:03.931636 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:03.931593    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:16:03.931709 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:03.931382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:03.931709 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:03.931642    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:16:03.931795 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:03.931757    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:16:04.110902 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:04.110871    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sv62g"]
Sep 11 18:16:04.114210 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:04.114188    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-65zrr"]
Sep 11 18:16:04.114890 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:04.114869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6jp5"]
Sep 11 18:16:04.135465 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:04.135442    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:16:04.135866 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:04.135555    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:16:04.135866 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:04.135441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:04.135866 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:04.135680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:04.135866 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:04.135709    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:16:04.135866 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:04.135784    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:16:04.135866 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:04.135815    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:16:04.150853 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:04.150829    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:16:05.139492 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:05.139460    2578 generic.go:356] "Generic (PLEG): container finished" podID="a2c0153c-5b15-490d-93cc-d06a4fd9c434" containerID="e0ebb09bf3a58153b92427264754bdb0f360d7621d66d801543e7e60de1fd471" exitCode=0
Sep 11 18:16:05.139908 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:05.139539    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhrj4" event={"ID":"a2c0153c-5b15-490d-93cc-d06a4fd9c434","Type":"ContainerDied","Data":"e0ebb09bf3a58153b92427264754bdb0f360d7621d66d801543e7e60de1fd471"}
Sep 11 18:16:05.931921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:05.931891    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:05.932045 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:05.931960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:16:05.932045 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:05.931967    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:16:05.932157 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:05.932060    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:16:05.932157 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:05.932106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:05.932246 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:05.932203    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:16:06.145250 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:06.145167    2578 generic.go:356] "Generic (PLEG): container finished" podID="a2c0153c-5b15-490d-93cc-d06a4fd9c434" containerID="23f6eb2fd419c3c95be4b4244ea8a850e1d3c3964fd0bdfff54da462b3fc7c97" exitCode=0
Sep 11 18:16:06.145250 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:06.145218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhrj4" event={"ID":"a2c0153c-5b15-490d-93cc-d06a4fd9c434","Type":"ContainerDied","Data":"23f6eb2fd419c3c95be4b4244ea8a850e1d3c3964fd0bdfff54da462b3fc7c97"}
Sep 11 18:16:07.931419 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:07.931387    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:16:07.931879 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:07.931387    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:07.931879 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:07.931500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:07.931879 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:07.931526    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:16:07.931879 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:07.931573    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:16:07.931879 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:07.931644    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:16:09.691800 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:09.691523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gp5\" (UniqueName: \"kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5\") pod \"network-check-target-g6jp5\" (UID: \"b0124408-a051-41e7-916e-ef1d331fc181\") " pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:09.692356 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:09.691842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:09.692356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.691710    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:16:09.692356 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:09.691879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:16:09.692356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.691915    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:16:09.692356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.691937    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f6gp5 for pod openshift-network-diagnostics/network-check-target-g6jp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:16:09.692356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.691998    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:16:09.692356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.692004    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5 podName:b0124408-a051-41e7-916e-ef1d331fc181 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:41.691985065 +0000 UTC m=+66.271249623 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6gp5" (UniqueName: "kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5") pod "network-check-target-g6jp5" (UID: "b0124408-a051-41e7-916e-ef1d331fc181") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:16:09.692356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.692049    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs podName:78c498ff-9632-4d5a-8dad-d1d9a94137e1 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:41.692031398 +0000 UTC m=+66.271295945 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs") pod "network-metrics-daemon-sv62g" (UID: "78c498ff-9632-4d5a-8dad-d1d9a94137e1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:16:09.692356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.692106    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:16:09.692356 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.692150    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret podName:2050106a-dd99-409d-9860-6887d827d3af nodeName:}" failed. No retries permitted until 2026-09-11 18:16:41.692139121 +0000 UTC m=+66.271403674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret") pod "global-pull-secret-syncer-65zrr" (UID: "2050106a-dd99-409d-9860-6887d827d3af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:16:09.931841 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:09.931807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:09.931841 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:09.931820    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:16:09.932063 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.931931    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65zrr" podUID="2050106a-dd99-409d-9860-6887d827d3af"
Sep 11 18:16:09.932063 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.931998    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:16:09.932063 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:09.932022    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:09.932223 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:09.932107    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6jp5" podUID="b0124408-a051-41e7-916e-ef1d331fc181"
Sep 11 18:16:10.218252 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.218226    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-186.ec2.internal" event="NodeReady"
Sep 11 18:16:10.218410 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.218349    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 18:16:10.267378 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.267353    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tjgtz"]
Sep 11 18:16:10.288441 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.288404    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kd5hz"]
Sep 11 18:16:10.288609 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.288540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.291362 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.291340    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 18:16:10.291512 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.291343    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 18:16:10.291512 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.291384    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksdw9\""
Sep 11 18:16:10.305902 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.305883    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kd5hz"]
Sep 11 18:16:10.306002 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.305908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tjgtz"]
Sep 11 18:16:10.306002 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.305991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kd5hz"
Sep 11 18:16:10.308607 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.308590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 18:16:10.308739 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.308723    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f8nfl\""
Sep 11 18:16:10.308791 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.308779    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 18:16:10.308853 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.308814    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 18:16:10.396545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.396515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6543db77-6046-4e91-9420-88dba053c7f3-config-volume\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.396545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.396549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79vh\" (UniqueName: \"kubernetes.io/projected/6543db77-6046-4e91-9420-88dba053c7f3-kube-api-access-z79vh\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.396759 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.396569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.396759 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.396590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d-cert\") pod \"ingress-canary-kd5hz\" (UID: \"7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d\") " pod="openshift-ingress-canary/ingress-canary-kd5hz"
Sep 11 18:16:10.396759 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.396727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6543db77-6046-4e91-9420-88dba053c7f3-tmp-dir\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.396856 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.396762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rw9z\" (UniqueName: \"kubernetes.io/projected/7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d-kube-api-access-6rw9z\") pod \"ingress-canary-kd5hz\" (UID: \"7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d\") " pod="openshift-ingress-canary/ingress-canary-kd5hz"
Sep 11 18:16:10.497636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.497605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6543db77-6046-4e91-9420-88dba053c7f3-tmp-dir\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.497768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.497685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rw9z\" (UniqueName: \"kubernetes.io/projected/7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d-kube-api-access-6rw9z\") pod \"ingress-canary-kd5hz\" (UID: \"7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d\") " pod="openshift-ingress-canary/ingress-canary-kd5hz"
Sep 11 18:16:10.497866 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.497836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6543db77-6046-4e91-9420-88dba053c7f3-config-volume\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.497961 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.497880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79vh\" (UniqueName: \"kubernetes.io/projected/6543db77-6046-4e91-9420-88dba053c7f3-kube-api-access-z79vh\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.497961 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.497906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.497961 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.497928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d-cert\") pod \"ingress-canary-kd5hz\" (UID: \"7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d\") " pod="openshift-ingress-canary/ingress-canary-kd5hz"
Sep 11 18:16:10.498116 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.497985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6543db77-6046-4e91-9420-88dba053c7f3-tmp-dir\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.498116 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:10.498096    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:10.498213 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:10.498163    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls podName:6543db77-6046-4e91-9420-88dba053c7f3 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:10.998145009 +0000 UTC m=+35.577409550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls") pod "dns-default-tjgtz" (UID: "6543db77-6046-4e91-9420-88dba053c7f3") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:10.498434 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.498413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6543db77-6046-4e91-9420-88dba053c7f3-config-volume\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.502461 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.502419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d-cert\") pod \"ingress-canary-kd5hz\" (UID: \"7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d\") " pod="openshift-ingress-canary/ingress-canary-kd5hz"
Sep 11 18:16:10.509214 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.509165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79vh\" (UniqueName: \"kubernetes.io/projected/6543db77-6046-4e91-9420-88dba053c7f3-kube-api-access-z79vh\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:10.509608 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.509581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rw9z\" (UniqueName: \"kubernetes.io/projected/7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d-kube-api-access-6rw9z\") pod \"ingress-canary-kd5hz\" (UID: \"7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d\") " pod="openshift-ingress-canary/ingress-canary-kd5hz"
Sep 11 18:16:10.614340 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.614316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kd5hz"
Sep 11 18:16:10.931961 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:10.931894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kd5hz"]
Sep 11 18:16:10.935149 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:16:10.934994    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d944d63_7ebc_46d4_8a4d_3b7d21e29c7d.slice/crio-34df5d5699fb931cc8c01fb4c5e893cd8b6106085f564f6a2da92bf20ca23631 WatchSource:0}: Error finding container 34df5d5699fb931cc8c01fb4c5e893cd8b6106085f564f6a2da92bf20ca23631: Status 404 returned error can't find the container with id 34df5d5699fb931cc8c01fb4c5e893cd8b6106085f564f6a2da92bf20ca23631
Sep 11 18:16:11.001692 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.001640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:11.001811 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:11.001796    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:11.001868 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:11.001859    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls podName:6543db77-6046-4e91-9420-88dba053c7f3 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:12.001842099 +0000 UTC m=+36.581106646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls") pod "dns-default-tjgtz" (UID: "6543db77-6046-4e91-9420-88dba053c7f3") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:11.158097 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.158061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kd5hz" event={"ID":"7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d","Type":"ContainerStarted","Data":"34df5d5699fb931cc8c01fb4c5e893cd8b6106085f564f6a2da92bf20ca23631"}
Sep 11 18:16:11.160791 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.160763    2578 generic.go:356] "Generic (PLEG): container finished" podID="a2c0153c-5b15-490d-93cc-d06a4fd9c434" containerID="104c2ced46ec80d7662c33459006fdfc2ea0344c4704a02f290994214eb096eb" exitCode=0
Sep 11 18:16:11.160895 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.160800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhrj4" event={"ID":"a2c0153c-5b15-490d-93cc-d06a4fd9c434","Type":"ContainerDied","Data":"104c2ced46ec80d7662c33459006fdfc2ea0344c4704a02f290994214eb096eb"}
Sep 11 18:16:11.931820 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.931735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:11.931820 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.931757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:16:11.932367 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.931735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:11.935989 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.935966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:16:11.936157 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.936058    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:16:11.936157 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.936098    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fhxbh\""
Sep 11 18:16:11.936157 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.936137    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:16:11.936327 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.936064    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kq7d5\""
Sep 11 18:16:11.936327 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:11.936205    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:16:12.008971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:12.008935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:12.009157 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:12.009117    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:12.009223 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:12.009186    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls podName:6543db77-6046-4e91-9420-88dba053c7f3 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:14.009166636 +0000 UTC m=+38.588431177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls") pod "dns-default-tjgtz" (UID: "6543db77-6046-4e91-9420-88dba053c7f3") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:12.166795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:12.166763    2578 generic.go:356] "Generic (PLEG): container finished" podID="a2c0153c-5b15-490d-93cc-d06a4fd9c434" containerID="94acf519377e4c37dc0151b2e65e7d6d0d300750a15b8c4629f99722844c7c4e" exitCode=0
Sep 11 18:16:12.166967 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:12.166827    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhrj4" event={"ID":"a2c0153c-5b15-490d-93cc-d06a4fd9c434","Type":"ContainerDied","Data":"94acf519377e4c37dc0151b2e65e7d6d0d300750a15b8c4629f99722844c7c4e"}
Sep 11 18:16:13.173209 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:13.173177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhrj4" event={"ID":"a2c0153c-5b15-490d-93cc-d06a4fd9c434","Type":"ContainerStarted","Data":"60a4697d5ab9852b5f5800dcaa760f27a35272e4ce616c1702cbb44d8a144c93"}
Sep 11 18:16:13.196018 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:13.195974    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hhrj4" podStartSLOduration=8.892350676 podStartE2EDuration="37.195959421s" totalImagesPullingTime="28.303608745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:13.194353902 +0000 UTC m=+37.773618463" watchObservedRunningTime="2026-09-11 18:16:13.195959421 +0000 UTC m=+37.775223982"
Sep 11 18:16:14.027238 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:14.027149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:14.027402 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:14.027256    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:14.027402 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:14.027309    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls podName:6543db77-6046-4e91-9420-88dba053c7f3 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:18.027295582 +0000 UTC m=+42.606560127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls") pod "dns-default-tjgtz" (UID: "6543db77-6046-4e91-9420-88dba053c7f3") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:14.176615 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:14.176570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kd5hz" event={"ID":"7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d","Type":"ContainerStarted","Data":"06d403bb97405a93e21b7407438fe36844b0c7db9a683b70bbcf290146c6bf5a"}
Sep 11 18:16:14.193072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:14.193021    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kd5hz" podStartSLOduration=1.976522761 podStartE2EDuration="4.1930023s" totalImagesPullingTime="2.216479539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:14.192447063 +0000 UTC m=+38.771711636" watchObservedRunningTime="2026-09-11 18:16:14.1930023 +0000 UTC m=+38.772266877"
Sep 11 18:16:18.052826 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:18.052618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:18.053215 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:18.052758    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:18.053215 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:18.052910    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls podName:6543db77-6046-4e91-9420-88dba053c7f3 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:26.05289521 +0000 UTC m=+50.632159752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls") pod "dns-default-tjgtz" (UID: "6543db77-6046-4e91-9420-88dba053c7f3") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:26.103427 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:26.103387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:26.103912 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:26.103496    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:26.103912 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:26.103544    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls podName:6543db77-6046-4e91-9420-88dba053c7f3 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.103531061 +0000 UTC m=+66.682795614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls") pod "dns-default-tjgtz" (UID: "6543db77-6046-4e91-9420-88dba053c7f3") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:36.155856 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:36.155827    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfrlp"
Sep 11 18:16:41.711324 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.711282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:41.711324 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.711333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:16:41.711819 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.711535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gp5\" (UniqueName: \"kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5\") pod \"network-check-target-g6jp5\" (UID: \"b0124408-a051-41e7-916e-ef1d331fc181\") " pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:41.714483 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.714455    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:16:41.714629 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.714585    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:16:41.714717 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.714668    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:16:41.721935 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:41.721918    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 18:16:41.721999 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:41.721985    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs podName:78c498ff-9632-4d5a-8dad-d1d9a94137e1 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:45.721965913 +0000 UTC m=+130.301230451 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs") pod "network-metrics-daemon-sv62g" (UID: "78c498ff-9632-4d5a-8dad-d1d9a94137e1") : secret "metrics-daemon-secret" not found
Sep 11 18:16:41.724032 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.724014    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:16:41.725313 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.725287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2050106a-dd99-409d-9860-6887d827d3af-original-pull-secret\") pod \"global-pull-secret-syncer-65zrr\" (UID: \"2050106a-dd99-409d-9860-6887d827d3af\") " pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:41.734044 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.734024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gp5\" (UniqueName: \"kubernetes.io/projected/b0124408-a051-41e7-916e-ef1d331fc181-kube-api-access-f6gp5\") pod \"network-check-target-g6jp5\" (UID: \"b0124408-a051-41e7-916e-ef1d331fc181\") " pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:41.947991 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.947964    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fhxbh\""
Sep 11 18:16:41.955144 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.955126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:41.958744 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:41.958727    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65zrr"
Sep 11 18:16:42.114534 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:42.114498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:16:42.114718 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:42.114640    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:42.114766 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:16:42.114726    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls podName:6543db77-6046-4e91-9420-88dba053c7f3 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:14.11470879 +0000 UTC m=+98.693973328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls") pod "dns-default-tjgtz" (UID: "6543db77-6046-4e91-9420-88dba053c7f3") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:42.225701 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:42.225617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6jp5"]
Sep 11 18:16:42.228932 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:16:42.228902    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0124408_a051_41e7_916e_ef1d331fc181.slice/crio-e634c3e506b57e769281b44e121df5a4c81d79e14540d140b70ff89c4dda3820 WatchSource:0}: Error finding container e634c3e506b57e769281b44e121df5a4c81d79e14540d140b70ff89c4dda3820: Status 404 returned error can't find the container with id e634c3e506b57e769281b44e121df5a4c81d79e14540d140b70ff89c4dda3820
Sep 11 18:16:42.229141 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:42.229122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-65zrr"]
Sep 11 18:16:42.232276 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:16:42.232255    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2050106a_dd99_409d_9860_6887d827d3af.slice/crio-ea2d18dff27f81a4ca253baf3f16add4388c23aa47cc7073c46c7ae1bdfb0a0a WatchSource:0}: Error finding container ea2d18dff27f81a4ca253baf3f16add4388c23aa47cc7073c46c7ae1bdfb0a0a: Status 404 returned error can't find the container with id ea2d18dff27f81a4ca253baf3f16add4388c23aa47cc7073c46c7ae1bdfb0a0a
Sep 11 18:16:42.243407 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:42.243366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6jp5" event={"ID":"b0124408-a051-41e7-916e-ef1d331fc181","Type":"ContainerStarted","Data":"e634c3e506b57e769281b44e121df5a4c81d79e14540d140b70ff89c4dda3820"}
Sep 11 18:16:42.244619 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:42.244597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-65zrr" event={"ID":"2050106a-dd99-409d-9860-6887d827d3af","Type":"ContainerStarted","Data":"ea2d18dff27f81a4ca253baf3f16add4388c23aa47cc7073c46c7ae1bdfb0a0a"}
Sep 11 18:16:47.258876 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:47.258841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-65zrr" event={"ID":"2050106a-dd99-409d-9860-6887d827d3af","Type":"ContainerStarted","Data":"340c33947d274b5ee390e3141cf3fa8c2e8d5490c81b304f51cb3a6809289387"}
Sep 11 18:16:47.260295 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:47.260271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6jp5" event={"ID":"b0124408-a051-41e7-916e-ef1d331fc181","Type":"ContainerStarted","Data":"78daa1f6dfa6d7298f1d56966927d39b44c82ec17abd55faf119af3ddbb67ebd"}
Sep 11 18:16:47.260411 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:47.260400    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:16:47.276050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:47.276005    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-65zrr" podStartSLOduration=66.937402685 podStartE2EDuration="1m11.275988711s" totalImagesPullingTime="4.338586026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:47.275845933 +0000 UTC m=+71.855110494" watchObservedRunningTime="2026-09-11 18:16:47.275988711 +0000 UTC m=+71.855253272"
Sep 11 18:16:47.294444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:16:47.294372    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g6jp5" podStartSLOduration=66.956098916 podStartE2EDuration="1m11.294358831s" totalImagesPullingTime="4.338259915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:47.293919081 +0000 UTC m=+71.873183642" watchObservedRunningTime="2026-09-11 18:16:47.294358831 +0000 UTC m=+71.873623391"
Sep 11 18:17:14.122750 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:14.122720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:17:14.123140 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:14.122831    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:17:14.123140 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:14.122890    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls podName:6543db77-6046-4e91-9420-88dba053c7f3 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:18.122876766 +0000 UTC m=+162.702141304 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls") pod "dns-default-tjgtz" (UID: "6543db77-6046-4e91-9420-88dba053c7f3") : secret "dns-default-metrics-tls" not found
Sep 11 18:17:18.265715 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:18.265685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g6jp5"
Sep 11 18:17:44.363351 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.363320    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ccz46"]
Sep 11 18:17:44.365200 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.365182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ccz46"
Sep 11 18:17:44.367807 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.367784    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dpvdt\""
Sep 11 18:17:44.374493 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.374464    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ccz46"]
Sep 11 18:17:44.416244 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.416219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vptmf\" (UniqueName: \"kubernetes.io/projected/ee2d7ba9-c609-4ed8-b2c9-bef2c53be44a-kube-api-access-vptmf\") pod \"network-check-source-6f8b768455-ccz46\" (UID: \"ee2d7ba9-c609-4ed8-b2c9-bef2c53be44a\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ccz46"
Sep 11 18:17:44.516661 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.516621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptmf\" (UniqueName: \"kubernetes.io/projected/ee2d7ba9-c609-4ed8-b2c9-bef2c53be44a-kube-api-access-vptmf\") pod \"network-check-source-6f8b768455-ccz46\" (UID: \"ee2d7ba9-c609-4ed8-b2c9-bef2c53be44a\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ccz46"
Sep 11 18:17:44.533748 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.533721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vptmf\" (UniqueName: \"kubernetes.io/projected/ee2d7ba9-c609-4ed8-b2c9-bef2c53be44a-kube-api-access-vptmf\") pod \"network-check-source-6f8b768455-ccz46\" (UID: \"ee2d7ba9-c609-4ed8-b2c9-bef2c53be44a\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ccz46"
Sep 11 18:17:44.564092 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.564070    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d"]
Sep 11 18:17:44.566588 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.566574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d"
Sep 11 18:17:44.571693 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.571674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ldr8b\""
Sep 11 18:17:44.571787 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.571740    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 18:17:44.572097 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.572078    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:17:44.572499 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.572482    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cfd5754c-j45cz"]
Sep 11 18:17:44.574326 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.574309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.578160 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.578141    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 18:17:44.578318 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.578262    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 18:17:44.578592 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.578575    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-grmtt\""
Sep 11 18:17:44.578886 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.578866    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 18:17:44.579032 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.578998    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d"]
Sep 11 18:17:44.584325 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.584307    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 18:17:44.596093 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.596072    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cfd5754c-j45cz"]
Sep 11 18:17:44.616953 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.616899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.616953 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.616931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-ca-trust-extracted\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.616953 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.616947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-trusted-ca\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.617129 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.616986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-certificates\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.617129 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.617003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-bound-sa-token\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.617129 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.617019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jqpm\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-kube-api-access-4jqpm\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.617129 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.617043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758k6\" (UniqueName: \"kubernetes.io/projected/ac502d25-97f4-4c61-9725-93b13caf9437-kube-api-access-758k6\") pod \"volume-data-source-validator-86695b7cc7-4429d\" (UID: \"ac502d25-97f4-4c61-9725-93b13caf9437\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d"
Sep 11 18:17:44.617129 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.617090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-installation-pull-secrets\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.617306 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.617130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-image-registry-private-configuration\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.673271 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.673252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ccz46"
Sep 11 18:17:44.717497 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.717466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.717613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.717509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-ca-trust-extracted\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.717613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.717536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-trusted-ca\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.717613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.717591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-certificates\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.717853 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.717614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-bound-sa-token\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.717853 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.717644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jqpm\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-kube-api-access-4jqpm\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.717853 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:44.717680    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:17:44.717853 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:44.717704    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfd5754c-j45cz: secret "image-registry-tls" not found
Sep 11 18:17:44.717853 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.717712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758k6\" (UniqueName: \"kubernetes.io/projected/ac502d25-97f4-4c61-9725-93b13caf9437-kube-api-access-758k6\") pod \"volume-data-source-validator-86695b7cc7-4429d\" (UID: \"ac502d25-97f4-4c61-9725-93b13caf9437\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d"
Sep 11 18:17:44.717853 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.717740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-installation-pull-secrets\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.717853 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.717769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-image-registry-private-configuration\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.717853 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:44.717786    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls podName:e0f59d9a-3c6e-49cd-a56e-9df59f10722a nodeName:}" failed. No retries permitted until 2026-09-11 18:17:45.217763343 +0000 UTC m=+129.797027882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls") pod "image-registry-5cfd5754c-j45cz" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a") : secret "image-registry-tls" not found
Sep 11 18:17:44.718855 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.718601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-trusted-ca\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.718855 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.718729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-certificates\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.719151 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.718866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-ca-trust-extracted\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.721211 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.721151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-image-registry-private-configuration\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.721319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.721282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-installation-pull-secrets\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.728795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.728772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jqpm\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-kube-api-access-4jqpm\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.728904 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.728770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-bound-sa-token\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:44.729098 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.729078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-758k6\" (UniqueName: \"kubernetes.io/projected/ac502d25-97f4-4c61-9725-93b13caf9437-kube-api-access-758k6\") pod \"volume-data-source-validator-86695b7cc7-4429d\" (UID: \"ac502d25-97f4-4c61-9725-93b13caf9437\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d"
Sep 11 18:17:44.874937 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.874883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d"
Sep 11 18:17:44.938591 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:44.938566    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ccz46"]
Sep 11 18:17:44.941320 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:17:44.941295    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2d7ba9_c609_4ed8_b2c9_bef2c53be44a.slice/crio-6427e9582df42c20a8d884b39e8f6da187294c86393932a6bb932a90d6209ff3 WatchSource:0}: Error finding container 6427e9582df42c20a8d884b39e8f6da187294c86393932a6bb932a90d6209ff3: Status 404 returned error can't find the container with id 6427e9582df42c20a8d884b39e8f6da187294c86393932a6bb932a90d6209ff3
Sep 11 18:17:45.138467 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:45.138402    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d"]
Sep 11 18:17:45.141442 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:17:45.141421    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac502d25_97f4_4c61_9725_93b13caf9437.slice/crio-db4b591d5a51adc2b437151a4987b4f040bbdb0ef91955e2ecf04ea44b5eda1d WatchSource:0}: Error finding container db4b591d5a51adc2b437151a4987b4f040bbdb0ef91955e2ecf04ea44b5eda1d: Status 404 returned error can't find the container with id db4b591d5a51adc2b437151a4987b4f040bbdb0ef91955e2ecf04ea44b5eda1d
Sep 11 18:17:45.221002 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:45.220977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:45.221091 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:45.221079    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:17:45.221091 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:45.221090    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfd5754c-j45cz: secret "image-registry-tls" not found
Sep 11 18:17:45.221162 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:45.221129    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls podName:e0f59d9a-3c6e-49cd-a56e-9df59f10722a nodeName:}" failed. No retries permitted until 2026-09-11 18:17:46.221117881 +0000 UTC m=+130.800382419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls") pod "image-registry-5cfd5754c-j45cz" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a") : secret "image-registry-tls" not found
Sep 11 18:17:45.393485 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:45.393425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d" event={"ID":"ac502d25-97f4-4c61-9725-93b13caf9437","Type":"ContainerStarted","Data":"db4b591d5a51adc2b437151a4987b4f040bbdb0ef91955e2ecf04ea44b5eda1d"}
Sep 11 18:17:45.394930 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:45.394905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ccz46" event={"ID":"ee2d7ba9-c609-4ed8-b2c9-bef2c53be44a","Type":"ContainerStarted","Data":"52dae46acf99aa495674392415b2d24942eb81306133c609d75bcb423682639a"}
Sep 11 18:17:45.395028 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:45.394950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ccz46" event={"ID":"ee2d7ba9-c609-4ed8-b2c9-bef2c53be44a","Type":"ContainerStarted","Data":"6427e9582df42c20a8d884b39e8f6da187294c86393932a6bb932a90d6209ff3"}
Sep 11 18:17:45.410275 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:45.410237    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ccz46" podStartSLOduration=1.410223427 podStartE2EDuration="1.410223427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:45.410015801 +0000 UTC m=+129.989280358" watchObservedRunningTime="2026-09-11 18:17:45.410223427 +0000 UTC m=+129.989487983"
Sep 11 18:17:45.724102 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:45.724033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:17:45.724223 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:45.724161    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 18:17:45.724291 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:45.724225    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs podName:78c498ff-9632-4d5a-8dad-d1d9a94137e1 nodeName:}" failed. No retries permitted until 2026-09-11 18:19:47.724207341 +0000 UTC m=+252.303471892 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs") pod "network-metrics-daemon-sv62g" (UID: "78c498ff-9632-4d5a-8dad-d1d9a94137e1") : secret "metrics-daemon-secret" not found
Sep 11 18:17:46.227376 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:46.227342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:46.227507 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:46.227465    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:17:46.227507 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:46.227477    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfd5754c-j45cz: secret "image-registry-tls" not found
Sep 11 18:17:46.227577 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:46.227525    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls podName:e0f59d9a-3c6e-49cd-a56e-9df59f10722a nodeName:}" failed. No retries permitted until 2026-09-11 18:17:48.227509425 +0000 UTC m=+132.806773977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls") pod "image-registry-5cfd5754c-j45cz" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a") : secret "image-registry-tls" not found
Sep 11 18:17:46.399210 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:46.399117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d" event={"ID":"ac502d25-97f4-4c61-9725-93b13caf9437","Type":"ContainerStarted","Data":"43718e822b7e50348f0de8ae4947ab9a2760123ce3a995fde1d1238626b46de8"}
Sep 11 18:17:46.415368 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:46.415324    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4429d" podStartSLOduration=1.45728065 podStartE2EDuration="2.415310916s" totalImagesPullingTime="958.030266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:46.415089365 +0000 UTC m=+130.994353924" watchObservedRunningTime="2026-09-11 18:17:46.415310916 +0000 UTC m=+130.994575475"
Sep 11 18:17:48.243791 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:48.243759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:48.244155 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:48.243907    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:17:48.244155 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:48.243925    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfd5754c-j45cz: secret "image-registry-tls" not found
Sep 11 18:17:48.244155 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:48.243988    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls podName:e0f59d9a-3c6e-49cd-a56e-9df59f10722a nodeName:}" failed. No retries permitted until 2026-09-11 18:17:52.243972069 +0000 UTC m=+136.823236616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls") pod "image-registry-5cfd5754c-j45cz" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a") : secret "image-registry-tls" not found
Sep 11 18:17:49.295635 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.295602    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-578v7"]
Sep 11 18:17:49.297687 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.297666    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.300313 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.300292    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 18:17:49.300417 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.300343    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 18:17:49.300417 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.300350    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-psjlx\""
Sep 11 18:17:49.301377 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.301362    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 18:17:49.301446 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.301363    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 18:17:49.308511 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.308491    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-578v7"]
Sep 11 18:17:49.350322 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.350303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/509846c6-e854-4e12-be62-b604e7b76faa-crio-socket\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.350400 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.350351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/509846c6-e854-4e12-be62-b604e7b76faa-data-volume\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.350400 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.350374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzr9\" (UniqueName: \"kubernetes.io/projected/509846c6-e854-4e12-be62-b604e7b76faa-kube-api-access-5jzr9\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.350400 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.350392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.350504 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.350432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.451220 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.451198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/509846c6-e854-4e12-be62-b604e7b76faa-crio-socket\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.451287 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.451243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/509846c6-e854-4e12-be62-b604e7b76faa-data-volume\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.451287 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.451260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzr9\" (UniqueName: \"kubernetes.io/projected/509846c6-e854-4e12-be62-b604e7b76faa-kube-api-access-5jzr9\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.451287 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.451277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.451388 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.451303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.451388 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.451309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/509846c6-e854-4e12-be62-b604e7b76faa-crio-socket\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.451388 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:49.451370    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:17:49.451388 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:49.451380    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:17:49.451499 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:49.451427    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:17:49.951415697 +0000 UTC m=+134.530680235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:17:49.451499 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:49.451439    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:17:49.951432883 +0000 UTC m=+134.530697421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : configmap "kube-rbac-proxy" not found
Sep 11 18:17:49.451597 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.451569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/509846c6-e854-4e12-be62-b604e7b76faa-data-volume\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.461507 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.461484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzr9\" (UniqueName: \"kubernetes.io/projected/509846c6-e854-4e12-be62-b604e7b76faa-kube-api-access-5jzr9\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.953714 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.953688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.953921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:49.953729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:49.953921 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:49.953806    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:17:49.953921 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:49.953835    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:17:49.953921 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:49.953876    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:17:50.953864486 +0000 UTC m=+135.533129024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:17:49.953921 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:49.953888    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:17:50.953882412 +0000 UTC m=+135.533146950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : configmap "kube-rbac-proxy" not found
Sep 11 18:17:50.960938 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:50.960901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:50.961303 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:50.960973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:50.961303 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:50.961045    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:17:50.961303 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:50.961048    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:17:50.961303 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:50.961096    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:17:52.961082094 +0000 UTC m=+137.540346646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : configmap "kube-rbac-proxy" not found
Sep 11 18:17:50.961303 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:50.961120    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:17:52.961103387 +0000 UTC m=+137.540367925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:17:51.276860 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.276829    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-l9r59"]
Sep 11 18:17:51.278979 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.278962    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.281756 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.281731    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 18:17:51.281868 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.281737    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 18:17:51.281868 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.281781    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 18:17:51.282830 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.282815    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 18:17:51.282918 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.282822    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-88r8s\""
Sep 11 18:17:51.293590 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.293570    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-l9r59"]
Sep 11 18:17:51.364406 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.364384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ecd4238e-6150-41cc-af9a-37cf736b87e3-signing-key\") pod \"service-ca-568869fdc5-l9r59\" (UID: \"ecd4238e-6150-41cc-af9a-37cf736b87e3\") " pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.364501 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.364417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcx54\" (UniqueName: \"kubernetes.io/projected/ecd4238e-6150-41cc-af9a-37cf736b87e3-kube-api-access-dcx54\") pod \"service-ca-568869fdc5-l9r59\" (UID: \"ecd4238e-6150-41cc-af9a-37cf736b87e3\") " pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.364501 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.364462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ecd4238e-6150-41cc-af9a-37cf736b87e3-signing-cabundle\") pod \"service-ca-568869fdc5-l9r59\" (UID: \"ecd4238e-6150-41cc-af9a-37cf736b87e3\") " pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.466129 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.466108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ecd4238e-6150-41cc-af9a-37cf736b87e3-signing-cabundle\") pod \"service-ca-568869fdc5-l9r59\" (UID: \"ecd4238e-6150-41cc-af9a-37cf736b87e3\") " pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.466217 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.466153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ecd4238e-6150-41cc-af9a-37cf736b87e3-signing-key\") pod \"service-ca-568869fdc5-l9r59\" (UID: \"ecd4238e-6150-41cc-af9a-37cf736b87e3\") " pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.466217 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.466176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcx54\" (UniqueName: \"kubernetes.io/projected/ecd4238e-6150-41cc-af9a-37cf736b87e3-kube-api-access-dcx54\") pod \"service-ca-568869fdc5-l9r59\" (UID: \"ecd4238e-6150-41cc-af9a-37cf736b87e3\") " pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.466815 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.466795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ecd4238e-6150-41cc-af9a-37cf736b87e3-signing-cabundle\") pod \"service-ca-568869fdc5-l9r59\" (UID: \"ecd4238e-6150-41cc-af9a-37cf736b87e3\") " pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.468421 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.468405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ecd4238e-6150-41cc-af9a-37cf736b87e3-signing-key\") pod \"service-ca-568869fdc5-l9r59\" (UID: \"ecd4238e-6150-41cc-af9a-37cf736b87e3\") " pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.474976 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.474949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcx54\" (UniqueName: \"kubernetes.io/projected/ecd4238e-6150-41cc-af9a-37cf736b87e3-kube-api-access-dcx54\") pod \"service-ca-568869fdc5-l9r59\" (UID: \"ecd4238e-6150-41cc-af9a-37cf736b87e3\") " pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.587444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.587396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-l9r59"
Sep 11 18:17:51.852846 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.852779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-l9r59"]
Sep 11 18:17:51.853323 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:51.853304    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2x9c6_101bbe73-40f7-41b8-89a2-b09ba7ef75ca/dns-node-resolver/0.log"
Sep 11 18:17:51.855611 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:17:51.855587    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd4238e_6150_41cc_af9a_37cf736b87e3.slice/crio-d67407ad9fb87a7c762198f7dd5035dede31baa85e5b06c1ab1da39c02dd4e26 WatchSource:0}: Error finding container d67407ad9fb87a7c762198f7dd5035dede31baa85e5b06c1ab1da39c02dd4e26: Status 404 returned error can't find the container with id d67407ad9fb87a7c762198f7dd5035dede31baa85e5b06c1ab1da39c02dd4e26
Sep 11 18:17:52.272691 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:52.272668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:17:52.273115 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:52.272806    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:17:52.273115 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:52.272826    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfd5754c-j45cz: secret "image-registry-tls" not found
Sep 11 18:17:52.273115 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:52.272887    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls podName:e0f59d9a-3c6e-49cd-a56e-9df59f10722a nodeName:}" failed. No retries permitted until 2026-09-11 18:18:00.272869843 +0000 UTC m=+144.852134385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls") pod "image-registry-5cfd5754c-j45cz" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a") : secret "image-registry-tls" not found
Sep 11 18:17:52.413199 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:52.413168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-l9r59" event={"ID":"ecd4238e-6150-41cc-af9a-37cf736b87e3","Type":"ContainerStarted","Data":"d67407ad9fb87a7c762198f7dd5035dede31baa85e5b06c1ab1da39c02dd4e26"}
Sep 11 18:17:52.978429 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:52.978392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:52.978672 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:52.978454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:52.978672 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:52.978548    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:17:52.978672 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:52.978597    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:17:52.978672 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:52.978613    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:17:56.978597926 +0000 UTC m=+141.557862469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : configmap "kube-rbac-proxy" not found
Sep 11 18:17:52.978672 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:52.978663    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:17:56.978633287 +0000 UTC m=+141.557897840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:17:53.053251 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:53.053229    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l4h5q_861c7885-6882-4d54-a19a-99b7c86dc58e/node-ca/0.log"
Sep 11 18:17:54.056156 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:54.056131    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kd5hz_7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d/serve-healthcheck-canary/0.log"
Sep 11 18:17:54.421338 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:54.421270    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-l9r59" event={"ID":"ecd4238e-6150-41cc-af9a-37cf736b87e3","Type":"ContainerStarted","Data":"153828b6c84ae172160f710a05da41a099d0eee4f2b0737427a5ecc1b6d926ae"}
Sep 11 18:17:54.437954 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:54.437910    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-l9r59" podStartSLOduration=1.699461582 podStartE2EDuration="3.437894865s" totalImagesPullingTime="1.738433283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:54.437110915 +0000 UTC m=+139.016375474" watchObservedRunningTime="2026-09-11 18:17:54.437894865 +0000 UTC m=+139.017159440"
Sep 11 18:17:57.011466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:57.011426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:57.011466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:17:57.011480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:17:57.011885 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:57.011542    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:17:57.011885 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:57.011593    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:17:57.011885 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:57.011617    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:18:05.011602397 +0000 UTC m=+149.590866935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : configmap "kube-rbac-proxy" not found
Sep 11 18:17:57.011885 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:17:57.011639    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:18:05.011628878 +0000 UTC m=+149.590893435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:00.335728 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:00.335690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:18:00.338057 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:00.338030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls\") pod \"image-registry-5cfd5754c-j45cz\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") " pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:18:00.483571 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:00.483540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:18:00.756709 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:00.756622    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cfd5754c-j45cz"]
Sep 11 18:18:00.759820 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:18:00.759792    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f59d9a_3c6e_49cd_a56e_9df59f10722a.slice/crio-d18568885f38b2390d2837e13f8e6be603c0aa93c40b4a7ddce4040ae03dd4cd WatchSource:0}: Error finding container d18568885f38b2390d2837e13f8e6be603c0aa93c40b4a7ddce4040ae03dd4cd: Status 404 returned error can't find the container with id d18568885f38b2390d2837e13f8e6be603c0aa93c40b4a7ddce4040ae03dd4cd
Sep 11 18:18:01.444552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:01.444518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz" event={"ID":"e0f59d9a-3c6e-49cd-a56e-9df59f10722a","Type":"ContainerStarted","Data":"2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010"}
Sep 11 18:18:01.444907 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:01.444566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz" event={"ID":"e0f59d9a-3c6e-49cd-a56e-9df59f10722a","Type":"ContainerStarted","Data":"d18568885f38b2390d2837e13f8e6be603c0aa93c40b4a7ddce4040ae03dd4cd"}
Sep 11 18:18:01.444907 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:01.444636    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:18:01.473795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:01.473744    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz" podStartSLOduration=17.473726589 podStartE2EDuration="17.473726589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:01.47267138 +0000 UTC m=+146.051935942" watchObservedRunningTime="2026-09-11 18:18:01.473726589 +0000 UTC m=+146.052991149"
Sep 11 18:18:05.073426 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:05.073393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:18:05.073785 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:05.073441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:18:05.073785 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:05.073537    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:05.073785 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:05.073590    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls podName:509846c6-e854-4e12-be62-b604e7b76faa nodeName:}" failed. No retries permitted until 2026-09-11 18:18:21.073577075 +0000 UTC m=+165.652841614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-578v7" (UID: "509846c6-e854-4e12-be62-b604e7b76faa") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:05.073959 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:05.073939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/509846c6-e854-4e12-be62-b604e7b76faa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:18:10.627787 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.627757    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cfd5754c-j45cz"]
Sep 11 18:18:10.634867 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.634836    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"]
Sep 11 18:18:10.638011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.637992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.641285 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:10.641257    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret"
Sep 11 18:18:10.641416 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:10.641305    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-service-proxy-server-certificates\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret"
Sep 11 18:18:10.641416 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:10.641315    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 11 18:18:10.641851 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:10.641813    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret"
Sep 11 18:18:10.641851 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:10.641815    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret"
Sep 11 18:18:10.642028 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:10.641843    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-ca\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret"
Sep 11 18:18:10.642367 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:10.642329    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 11 18:18:10.657882 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.657860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"]
Sep 11 18:18:10.814197 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.814172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.814347 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.814213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfr4t\" (UniqueName: \"kubernetes.io/projected/cd42d551-28f2-4f7a-aabf-66d255d4be9d-kube-api-access-mfr4t\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.814347 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.814243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd42d551-28f2-4f7a-aabf-66d255d4be9d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.814347 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.814326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-hub\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.814445 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.814365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-ca\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.814445 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.814381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.915390 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.915315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd42d551-28f2-4f7a-aabf-66d255d4be9d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.915390 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.915362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-hub\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.915390 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.915389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-ca\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.915597 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.915409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.915597 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.915445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:10.915597 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:10.915485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfr4t\" (UniqueName: \"kubernetes.io/projected/cd42d551-28f2-4f7a-aabf-66d255d4be9d-kube-api-access-mfr4t\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:11.602939 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:11.602906    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 18:18:11.608508 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:11.608484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-hub\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:11.684325 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:11.684296    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 18:18:11.686868 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:11.686851    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd42d551-28f2-4f7a-aabf-66d255d4be9d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:11.725227 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:11.725209    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 18:18:11.728250 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:11.728219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:11.915919 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:11.915842    2578 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-ca: failed to sync secret cache: timed out waiting for the condition
Sep 11 18:18:11.916033 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:11.915928    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-ca podName:cd42d551-28f2-4f7a-aabf-66d255d4be9d nodeName:}" failed. No retries permitted until 2026-09-11 18:18:12.415912273 +0000 UTC m=+156.995176827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ca" (UniqueName: "kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-ca") pod "cluster-proxy-proxy-agent-6dc8b497c5-f55wk" (UID: "cd42d551-28f2-4f7a-aabf-66d255d4be9d") : failed to sync secret cache: timed out waiting for the condition
Sep 11 18:18:11.916033 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:11.915842    2578 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-hub-kubeconfig: failed to sync secret cache: timed out waiting for the condition
Sep 11 18:18:11.916033 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:11.915983    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-hub-kubeconfig podName:cd42d551-28f2-4f7a-aabf-66d255d4be9d nodeName:}" failed. No retries permitted until 2026-09-11 18:18:12.415973192 +0000 UTC m=+156.995237732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "hub-kubeconfig" (UniqueName: "kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-hub-kubeconfig") pod "cluster-proxy-proxy-agent-6dc8b497c5-f55wk" (UID: "cd42d551-28f2-4f7a-aabf-66d255d4be9d") : failed to sync secret cache: timed out waiting for the condition
Sep 11 18:18:11.922117 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:11.922099    2578 projected.go:291] Couldn't get configMap open-cluster-management-agent-addon/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 11 18:18:11.922181 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:11.922125    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mfr4t for pod open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk: failed to sync configmap cache: timed out waiting for the condition
Sep 11 18:18:11.922181 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:11.922162    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd42d551-28f2-4f7a-aabf-66d255d4be9d-kube-api-access-mfr4t podName:cd42d551-28f2-4f7a-aabf-66d255d4be9d nodeName:}" failed. No retries permitted until 2026-09-11 18:18:12.422152763 +0000 UTC m=+157.001417301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mfr4t" (UniqueName: "kubernetes.io/projected/cd42d551-28f2-4f7a-aabf-66d255d4be9d-kube-api-access-mfr4t") pod "cluster-proxy-proxy-agent-6dc8b497c5-f55wk" (UID: "cd42d551-28f2-4f7a-aabf-66d255d4be9d") : failed to sync configmap cache: timed out waiting for the condition
Sep 11 18:18:11.987386 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:11.987358    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 18:18:12.003674 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.003640    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 18:18:12.069260 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.069235    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 18:18:12.207027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.206966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 18:18:12.426110 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.426083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:12.426253 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.426124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfr4t\" (UniqueName: \"kubernetes.io/projected/cd42d551-28f2-4f7a-aabf-66d255d4be9d-kube-api-access-mfr4t\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:12.426253 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.426175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-ca\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:12.428468 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.428446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-ca\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:12.428608 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.428588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfr4t\" (UniqueName: \"kubernetes.io/projected/cd42d551-28f2-4f7a-aabf-66d255d4be9d-kube-api-access-mfr4t\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:12.428682 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.428632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd42d551-28f2-4f7a-aabf-66d255d4be9d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6dc8b497c5-f55wk\" (UID: \"cd42d551-28f2-4f7a-aabf-66d255d4be9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:12.460554 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.460505    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"
Sep 11 18:18:12.726637 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:12.726567    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk"]
Sep 11 18:18:12.729478 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:18:12.729452    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd42d551_28f2_4f7a_aabf_66d255d4be9d.slice/crio-3e34e26e3bb3f9df47056908fc0632c6235017d67f7f1bcbae83eebeb25c71c3 WatchSource:0}: Error finding container 3e34e26e3bb3f9df47056908fc0632c6235017d67f7f1bcbae83eebeb25c71c3: Status 404 returned error can't find the container with id 3e34e26e3bb3f9df47056908fc0632c6235017d67f7f1bcbae83eebeb25c71c3
Sep 11 18:18:13.297958 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:13.297922    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tjgtz" podUID="6543db77-6046-4e91-9420-88dba053c7f3"
Sep 11 18:18:13.486094 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:13.486057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:18:13.486094 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:13.486056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk" event={"ID":"cd42d551-28f2-4f7a-aabf-66d255d4be9d","Type":"ContainerStarted","Data":"3e34e26e3bb3f9df47056908fc0632c6235017d67f7f1bcbae83eebeb25c71c3"}
Sep 11 18:18:14.952396 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:14.952358    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sv62g" podUID="78c498ff-9632-4d5a-8dad-d1d9a94137e1"
Sep 11 18:18:16.498316 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:16.498238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk" event={"ID":"cd42d551-28f2-4f7a-aabf-66d255d4be9d","Type":"ContainerStarted","Data":"c3f8b8a25c74e3cdd9efedd5b7a54c3fdc7a20bb0407f074007432e8755a8cb9"}
Sep 11 18:18:16.498316 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:16.498280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk" event={"ID":"cd42d551-28f2-4f7a-aabf-66d255d4be9d","Type":"ContainerStarted","Data":"9b300115f0890068096501f4bae4d05d1dbe2eb88db5595d125368c2182698ca"}
Sep 11 18:18:16.498316 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:16.498301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk" event={"ID":"cd42d551-28f2-4f7a-aabf-66d255d4be9d","Type":"ContainerStarted","Data":"7651f7978b2aef1c0a833b9567acb872b129f01a9b545463c9969c280a185cfe"}
Sep 11 18:18:16.517606 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:16.517563    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dc8b497c5-f55wk" podStartSLOduration=3.202621888 podStartE2EDuration="6.517551072s" totalImagesPullingTime="3.314929184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:16.516059858 +0000 UTC m=+161.095324456" watchObservedRunningTime="2026-09-11 18:18:16.517551072 +0000 UTC m=+161.096815631"
Sep 11 18:18:18.170824 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:18.170776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:18:18.172964 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:18.172947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6543db77-6046-4e91-9420-88dba053c7f3-metrics-tls\") pod \"dns-default-tjgtz\" (UID: \"6543db77-6046-4e91-9420-88dba053c7f3\") " pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:18:18.289755 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:18.289722    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksdw9\""
Sep 11 18:18:18.297111 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:18.297086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:18:18.567323 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:18.567291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tjgtz"]
Sep 11 18:18:18.570424 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:18:18.570395    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6543db77_6046_4e91_9420_88dba053c7f3.slice/crio-6adb785cea8f4e6eb5caae1bacf3e1a58848a21ddf4d5cb43f37e9d72fc7d511 WatchSource:0}: Error finding container 6adb785cea8f4e6eb5caae1bacf3e1a58848a21ddf4d5cb43f37e9d72fc7d511: Status 404 returned error can't find the container with id 6adb785cea8f4e6eb5caae1bacf3e1a58848a21ddf4d5cb43f37e9d72fc7d511
Sep 11 18:18:19.508995 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:19.508954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjgtz" event={"ID":"6543db77-6046-4e91-9420-88dba053c7f3","Type":"ContainerStarted","Data":"6adb785cea8f4e6eb5caae1bacf3e1a58848a21ddf4d5cb43f37e9d72fc7d511"}
Sep 11 18:18:20.513322 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:20.513284    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjgtz" event={"ID":"6543db77-6046-4e91-9420-88dba053c7f3","Type":"ContainerStarted","Data":"12ba1356fc33c27b23516ef30cf2f279580f40c1871025fe51f78fcc81bf22b6"}
Sep 11 18:18:20.513693 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:20.513361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjgtz" event={"ID":"6543db77-6046-4e91-9420-88dba053c7f3","Type":"ContainerStarted","Data":"0bf78c811ae4c6054b5e22b94bf52da7a29601e99ddd099cdd17ea80783c2aeb"}
Sep 11 18:18:20.513693 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:20.513403    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:18:20.531261 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:20.531212    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tjgtz" podStartSLOduration=129.165565799 podStartE2EDuration="2m10.53119717s" totalImagesPullingTime="1.365631371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:20.530696817 +0000 UTC m=+165.109961378" watchObservedRunningTime="2026-09-11 18:18:20.53119717 +0000 UTC m=+165.110461730"
Sep 11 18:18:20.632837 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:20.632803    2578 patch_prober.go:28] interesting pod/image-registry-5cfd5754c-j45cz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 18:18:20.632979 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:20.632856    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz" podUID="e0f59d9a-3c6e-49cd-a56e-9df59f10722a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:18:21.095483 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:21.095445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:18:21.097711 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:21.097679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/509846c6-e854-4e12-be62-b604e7b76faa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578v7\" (UID: \"509846c6-e854-4e12-be62-b604e7b76faa\") " pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:18:21.106565 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:21.106546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-578v7"
Sep 11 18:18:21.373289 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:21.373213    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-578v7"]
Sep 11 18:18:21.376169 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:18:21.376134    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509846c6_e854_4e12_be62_b604e7b76faa.slice/crio-2e81c9bfda5e49c84c9d27f5d8c183913e873bdb1256431038eca7d6f77ead5c WatchSource:0}: Error finding container 2e81c9bfda5e49c84c9d27f5d8c183913e873bdb1256431038eca7d6f77ead5c: Status 404 returned error can't find the container with id 2e81c9bfda5e49c84c9d27f5d8c183913e873bdb1256431038eca7d6f77ead5c
Sep 11 18:18:21.518466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:21.518430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-578v7" event={"ID":"509846c6-e854-4e12-be62-b604e7b76faa","Type":"ContainerStarted","Data":"1952ad916971446f1d3f8da143a57725718ca0a25d2b25c88f8a26594f20c31e"}
Sep 11 18:18:21.518837 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:21.518490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-578v7" event={"ID":"509846c6-e854-4e12-be62-b604e7b76faa","Type":"ContainerStarted","Data":"2e81c9bfda5e49c84c9d27f5d8c183913e873bdb1256431038eca7d6f77ead5c"}
Sep 11 18:18:22.524985 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:22.524953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-578v7" event={"ID":"509846c6-e854-4e12-be62-b604e7b76faa","Type":"ContainerStarted","Data":"b6e0c48c3629472297b6ff147616700356301e2dcc6c90baf9f5e6cc050b8835"}
Sep 11 18:18:23.530006 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:23.529969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-578v7" event={"ID":"509846c6-e854-4e12-be62-b604e7b76faa","Type":"ContainerStarted","Data":"6c3bd152ae8d316e254a5a8219f19e6e660a10a77d9e1625344dd316ad8b6fc4"}
Sep 11 18:18:23.550243 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:23.550196    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-578v7" podStartSLOduration=32.807527646 podStartE2EDuration="34.550179627s" totalImagesPullingTime="1.742651981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:17:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:23.549100906 +0000 UTC m=+168.128365478" watchObservedRunningTime="2026-09-11 18:18:23.550179627 +0000 UTC m=+168.129444187"
Sep 11 18:18:26.931734 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:26.931700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:18:30.520880 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:30.520811    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tjgtz"
Sep 11 18:18:30.631631 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:30.631603    2578 patch_prober.go:28] interesting pod/image-registry-5cfd5754c-j45cz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 18:18:30.631788 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:30.631662    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz" podUID="e0f59d9a-3c6e-49cd-a56e-9df59f10722a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:18:32.089224 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.089188    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"]
Sep 11 18:18:32.092301 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.092281    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.097769 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.097744    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 18:18:32.097858 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.097768    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 18:18:32.097906 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.097890    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 18:18:32.098003 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.097988    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 18:18:32.098093 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.098079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 18:18:32.098176 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.098162    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-j8tfl\""
Sep 11 18:18:32.105491 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.105469    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"]
Sep 11 18:18:32.169421 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.169396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjw4\" (UniqueName: \"kubernetes.io/projected/c35d6880-51f7-454f-a1bd-a1fe822a6f79-kube-api-access-9kjw4\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.169546 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.169465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c35d6880-51f7-454f-a1bd-a1fe822a6f79-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.169546 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.169495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c35d6880-51f7-454f-a1bd-a1fe822a6f79-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.169546 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.169516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c35d6880-51f7-454f-a1bd-a1fe822a6f79-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.270324 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.270296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c35d6880-51f7-454f-a1bd-a1fe822a6f79-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.270458 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.270333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c35d6880-51f7-454f-a1bd-a1fe822a6f79-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.270458 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.270357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c35d6880-51f7-454f-a1bd-a1fe822a6f79-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.270458 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.270376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjw4\" (UniqueName: \"kubernetes.io/projected/c35d6880-51f7-454f-a1bd-a1fe822a6f79-kube-api-access-9kjw4\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.271001 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.270979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c35d6880-51f7-454f-a1bd-a1fe822a6f79-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.272712 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.272690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c35d6880-51f7-454f-a1bd-a1fe822a6f79-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.272825 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.272803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c35d6880-51f7-454f-a1bd-a1fe822a6f79-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.279320 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.279295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjw4\" (UniqueName: \"kubernetes.io/projected/c35d6880-51f7-454f-a1bd-a1fe822a6f79-kube-api-access-9kjw4\") pod \"prometheus-operator-68bcd7f987-7tcxj\" (UID: \"c35d6880-51f7-454f-a1bd-a1fe822a6f79\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.408432 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.408362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"
Sep 11 18:18:32.673224 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:32.673153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj"]
Sep 11 18:18:32.675901 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:18:32.675878    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35d6880_51f7_454f_a1bd_a1fe822a6f79.slice/crio-5d10d0fa7c9b848dc0b242353d9263fbfaee26039268eda4e0c490ee7ba04789 WatchSource:0}: Error finding container 5d10d0fa7c9b848dc0b242353d9263fbfaee26039268eda4e0c490ee7ba04789: Status 404 returned error can't find the container with id 5d10d0fa7c9b848dc0b242353d9263fbfaee26039268eda4e0c490ee7ba04789
Sep 11 18:18:33.561906 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:33.561865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj" event={"ID":"c35d6880-51f7-454f-a1bd-a1fe822a6f79","Type":"ContainerStarted","Data":"5d10d0fa7c9b848dc0b242353d9263fbfaee26039268eda4e0c490ee7ba04789"}
Sep 11 18:18:34.566503 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:34.566469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj" event={"ID":"c35d6880-51f7-454f-a1bd-a1fe822a6f79","Type":"ContainerStarted","Data":"0fdf7028220f655c5e9b50530695c9d170c049fe36819765953287ac520bca1b"}
Sep 11 18:18:34.566894 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:34.566521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj" event={"ID":"c35d6880-51f7-454f-a1bd-a1fe822a6f79","Type":"ContainerStarted","Data":"f8bf4a0d1bd442989ffa952f4e89c13cd3f144a3eb7b05b408d1fc91353544e4"}
Sep 11 18:18:34.583474 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:34.583433    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-7tcxj" podStartSLOduration=1.4009382910000001 podStartE2EDuration="2.583416535s" totalImagesPullingTime="1.182478244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:34.583156463 +0000 UTC m=+179.162421022" watchObservedRunningTime="2026-09-11 18:18:34.583416535 +0000 UTC m=+179.162681095"
Sep 11 18:18:35.646994 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.646959    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz" podUID="e0f59d9a-3c6e-49cd-a56e-9df59f10722a" containerName="registry" containerID="cri-o://2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010" gracePeriod=30
Sep 11 18:18:35.884099 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.884077    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:18:35.995843 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.995770    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jqpm\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-kube-api-access-4jqpm\") pod \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") "
Sep 11 18:18:35.995843 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.995829    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls\") pod \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") "
Sep 11 18:18:35.996002 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.995850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-certificates\") pod \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") "
Sep 11 18:18:35.996002 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.995949    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-image-registry-private-configuration\") pod \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") "
Sep 11 18:18:35.996002 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.995991    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-trusted-ca\") pod \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") "
Sep 11 18:18:35.996135 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.996018    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-installation-pull-secrets\") pod \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") "
Sep 11 18:18:35.996135 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.996046    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-bound-sa-token\") pod \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") "
Sep 11 18:18:35.996135 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.996072    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-ca-trust-extracted\") pod \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\" (UID: \"e0f59d9a-3c6e-49cd-a56e-9df59f10722a\") "
Sep 11 18:18:35.996317 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.996191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e0f59d9a-3c6e-49cd-a56e-9df59f10722a" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:18:35.996453 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.996416    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e0f59d9a-3c6e-49cd-a56e-9df59f10722a" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:18:35.998182 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.998149    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e0f59d9a-3c6e-49cd-a56e-9df59f10722a" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:18:35.998182 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.998163    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e0f59d9a-3c6e-49cd-a56e-9df59f10722a" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:18:35.998328 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.998283    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-kube-api-access-4jqpm" (OuterVolumeSpecName: "kube-api-access-4jqpm") pod "e0f59d9a-3c6e-49cd-a56e-9df59f10722a" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a"). InnerVolumeSpecName "kube-api-access-4jqpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:18:35.998328 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.998294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e0f59d9a-3c6e-49cd-a56e-9df59f10722a" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:18:35.998447 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:35.998428    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e0f59d9a-3c6e-49cd-a56e-9df59f10722a" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:18:36.004440 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.004417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e0f59d9a-3c6e-49cd-a56e-9df59f10722a" (UID: "e0f59d9a-3c6e-49cd-a56e-9df59f10722a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:36.097066 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.097037    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-installation-pull-secrets\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:18:36.097066 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.097062    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-bound-sa-token\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:18:36.097205 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.097072    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-ca-trust-extracted\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:18:36.097205 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.097081    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jqpm\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-kube-api-access-4jqpm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:18:36.097205 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.097091    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-tls\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:18:36.097205 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.097098    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-registry-certificates\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:18:36.097205 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.097106    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-image-registry-private-configuration\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:18:36.097205 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.097115    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0f59d9a-3c6e-49cd-a56e-9df59f10722a-trusted-ca\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:18:36.544817 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.544780    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wlv42"]
Sep 11 18:18:36.545120 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.545106    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f59d9a-3c6e-49cd-a56e-9df59f10722a" containerName="registry"
Sep 11 18:18:36.545180 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.545123    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f59d9a-3c6e-49cd-a56e-9df59f10722a" containerName="registry"
Sep 11 18:18:36.545180 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.545177    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f59d9a-3c6e-49cd-a56e-9df59f10722a" containerName="registry"
Sep 11 18:18:36.550052 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.550031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.555070 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.555030    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 18:18:36.555224 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.555196    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 18:18:36.555315 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.555255    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 18:18:36.555437 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.555418    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kzmdz\""
Sep 11 18:18:36.574041 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.574018    2578 generic.go:356] "Generic (PLEG): container finished" podID="e0f59d9a-3c6e-49cd-a56e-9df59f10722a" containerID="2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010" exitCode=0
Sep 11 18:18:36.574144 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.574052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz" event={"ID":"e0f59d9a-3c6e-49cd-a56e-9df59f10722a","Type":"ContainerDied","Data":"2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010"}
Sep 11 18:18:36.574144 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.574065    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz"
Sep 11 18:18:36.574144 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.574087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfd5754c-j45cz" event={"ID":"e0f59d9a-3c6e-49cd-a56e-9df59f10722a","Type":"ContainerDied","Data":"d18568885f38b2390d2837e13f8e6be603c0aa93c40b4a7ddce4040ae03dd4cd"}
Sep 11 18:18:36.574144 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.574113    2578 scope.go:117] "RemoveContainer" containerID="2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010"
Sep 11 18:18:36.585510 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.585493    2578 scope.go:117] "RemoveContainer" containerID="2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010"
Sep 11 18:18:36.585821 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:36.585793    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010\": container with ID starting with 2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010 not found: ID does not exist" containerID="2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010"
Sep 11 18:18:36.585912 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.585833    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010"} err="failed to get container status \"2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010\": rpc error: code = NotFound desc = could not find container \"2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010\": container with ID starting with 2e230041187097ff049d8197066db77098213f3481da0bcea9fcd928e6d22010 not found: ID does not exist"
Sep 11 18:18:36.600523 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.600497    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cfd5754c-j45cz"]
Sep 11 18:18:36.600773 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.600753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-textfile\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.600822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.600787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-wtmp\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.600822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.600817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwsw\" (UniqueName: \"kubernetes.io/projected/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-kube-api-access-wkwsw\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.600938 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.600918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-metrics-client-ca\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.600997 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.600965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.600997 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.600989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-accelerators-collector-config\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.601069 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.601029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-sys\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.601069 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.601053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-root\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.601133 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.601076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-tls\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.612348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.612326    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cfd5754c-j45cz"]
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.701957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-textfile\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.701991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-wtmp\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.702018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwsw\" (UniqueName: \"kubernetes.io/projected/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-kube-api-access-wkwsw\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.702066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-metrics-client-ca\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.702101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.702126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-accelerators-collector-config\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.702159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-sys\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.702185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-root\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.702206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-tls\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.702284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-textfile\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:36.702361    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.702403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-wtmp\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:18:36.702413    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-tls podName:6fe2ae64-a407-4ecb-b22b-3e4ff72ca409 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:37.202395542 +0000 UTC m=+181.781660096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-tls") pod "node-exporter-wlv42" (UID: "6fe2ae64-a407-4ecb-b22b-3e4ff72ca409") : secret "node-exporter-tls" not found
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.703025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-metrics-client-ca\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.703087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-sys\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.703090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-accelerators-collector-config\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.703158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.703132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-root\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.705670 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.705638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:36.714491 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:36.714412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwsw\" (UniqueName: \"kubernetes.io/projected/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-kube-api-access-wkwsw\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:37.205437 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:37.205400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-tls\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:37.207813 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:37.207787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6fe2ae64-a407-4ecb-b22b-3e4ff72ca409-node-exporter-tls\") pod \"node-exporter-wlv42\" (UID: \"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409\") " pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:37.461136 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:37.461062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wlv42"
Sep 11 18:18:37.474240 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:18:37.474212    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe2ae64_a407_4ecb_b22b_3e4ff72ca409.slice/crio-a31361728f6ff9ac22d8e184f593e823c3783da024c116fbca1cc9d830d97f4c WatchSource:0}: Error finding container a31361728f6ff9ac22d8e184f593e823c3783da024c116fbca1cc9d830d97f4c: Status 404 returned error can't find the container with id a31361728f6ff9ac22d8e184f593e823c3783da024c116fbca1cc9d830d97f4c
Sep 11 18:18:37.580969 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:37.580934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wlv42" event={"ID":"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409","Type":"ContainerStarted","Data":"a31361728f6ff9ac22d8e184f593e823c3783da024c116fbca1cc9d830d97f4c"}
Sep 11 18:18:37.935767 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:37.935737    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0f59d9a-3c6e-49cd-a56e-9df59f10722a" path="/var/lib/kubelet/pods/e0f59d9a-3c6e-49cd-a56e-9df59f10722a/volumes"
Sep 11 18:18:38.447685 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.447663    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-588b5db496-jrr8p"]
Sep 11 18:18:38.451011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.450994    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.454131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.453861    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 18:18:38.454131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.453893    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 18:18:38.454131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.453925    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 18:18:38.454131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.453960    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 18:18:38.454131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.453995    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pgpnh\""
Sep 11 18:18:38.454131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.454026    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 18:18:38.454131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.453870    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f6aupvhbssfct\""
Sep 11 18:18:38.463099 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.463081    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-588b5db496-jrr8p"]
Sep 11 18:18:38.516642 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.516618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.516738 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.516674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.516738 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.516720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b7611c-a626-41d6-82f8-080815c87c27-metrics-client-ca\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.516817 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.516767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.516817 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.516794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxq4r\" (UniqueName: \"kubernetes.io/projected/d5b7611c-a626-41d6-82f8-080815c87c27-kube-api-access-jxq4r\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.516916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.516895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-tls\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.516949 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.516933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-grpc-tls\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.516981 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.516965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.589491 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.589462    2578 generic.go:356] "Generic (PLEG): container finished" podID="6fe2ae64-a407-4ecb-b22b-3e4ff72ca409" containerID="9f1e6d5e046d785cb9c0c21c47942f9fc92a75567779938a47d374dc399ebd1f" exitCode=0
Sep 11 18:18:38.589599 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.589515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wlv42" event={"ID":"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409","Type":"ContainerDied","Data":"9f1e6d5e046d785cb9c0c21c47942f9fc92a75567779938a47d374dc399ebd1f"}
Sep 11 18:18:38.617403 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.617378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.617494 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.617412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b7611c-a626-41d6-82f8-080815c87c27-metrics-client-ca\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.617494 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.617462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.617612 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.617586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxq4r\" (UniqueName: \"kubernetes.io/projected/d5b7611c-a626-41d6-82f8-080815c87c27-kube-api-access-jxq4r\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.617714 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.617700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-tls\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.617768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.617730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-grpc-tls\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.617768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.617758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.617873 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.617790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.618150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.618120    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b7611c-a626-41d6-82f8-080815c87c27-metrics-client-ca\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.620254 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.620235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.620254 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.620243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-tls\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.620370 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.620339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.620463 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.620445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.620680 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.620644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-grpc-tls\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.621374 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.621358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5b7611c-a626-41d6-82f8-080815c87c27-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.627179 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.627159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxq4r\" (UniqueName: \"kubernetes.io/projected/d5b7611c-a626-41d6-82f8-080815c87c27-kube-api-access-jxq4r\") pod \"thanos-querier-588b5db496-jrr8p\" (UID: \"d5b7611c-a626-41d6-82f8-080815c87c27\") " pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:38.776734 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:38.776703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:39.059142 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:39.059109    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-588b5db496-jrr8p"]
Sep 11 18:18:39.061771 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:18:39.061749    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b7611c_a626_41d6_82f8_080815c87c27.slice/crio-8b08d382dad64e659df7763b80764acc94c1cf0100bfde921e4ffe4ec7638db5 WatchSource:0}: Error finding container 8b08d382dad64e659df7763b80764acc94c1cf0100bfde921e4ffe4ec7638db5: Status 404 returned error can't find the container with id 8b08d382dad64e659df7763b80764acc94c1cf0100bfde921e4ffe4ec7638db5
Sep 11 18:18:39.594553 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:39.594518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wlv42" event={"ID":"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409","Type":"ContainerStarted","Data":"7791f8b01465af67d8ec67e765d5527e0a120b79e429d9a98a6538f0d4053c30"}
Sep 11 18:18:39.594758 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:39.594573    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wlv42" event={"ID":"6fe2ae64-a407-4ecb-b22b-3e4ff72ca409","Type":"ContainerStarted","Data":"419ef5829f5c51d770abada7fb1af7e759cfbca8e852abb9dcab3499f7b82de3"}
Sep 11 18:18:39.595767 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:39.595741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p" event={"ID":"d5b7611c-a626-41d6-82f8-080815c87c27","Type":"ContainerStarted","Data":"8b08d382dad64e659df7763b80764acc94c1cf0100bfde921e4ffe4ec7638db5"}
Sep 11 18:18:39.613223 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:39.613180    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wlv42" podStartSLOduration=2.592320888 podStartE2EDuration="3.613164437s" totalImagesPullingTime="960.95983ms" totalInitContainerRuntime="59.883719ms" isStatefulPod=true podCreationTimestamp="2026-09-11 18:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:39.612413994 +0000 UTC m=+184.191678553" watchObservedRunningTime="2026-09-11 18:18:39.613164437 +0000 UTC m=+184.192428997"
Sep 11 18:18:40.765000 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.764967    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"]
Sep 11 18:18:40.768277 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.768251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.771212 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.771191    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 18:18:40.771343 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.771295    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ch5umoefq1aih\""
Sep 11 18:18:40.772461 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.772446    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 18:18:40.772552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.772459    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 18:18:40.772606 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.772560    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tkr8f\""
Sep 11 18:18:40.772823 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.772809    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 18:18:40.780034 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.780013    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"]
Sep 11 18:18:40.837223 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.837190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-audit-log\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.837403 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.837233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-metrics-server-audit-profiles\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.837403 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.837329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-secret-metrics-server-tls\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.837403 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.837366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhvtk\" (UniqueName: \"kubernetes.io/projected/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-kube-api-access-bhvtk\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.837403 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.837397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-secret-metrics-server-client-certs\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.837615 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.837419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.837615 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.837464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-client-ca-bundle\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.938024 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.937985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-secret-metrics-server-client-certs\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.938215 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.938034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.938215 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.938173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-client-ca-bundle\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.938331 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.938216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-audit-log\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.938331 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.938247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-metrics-server-audit-profiles\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.938331 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.938318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-secret-metrics-server-tls\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.938473 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.938353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhvtk\" (UniqueName: \"kubernetes.io/projected/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-kube-api-access-bhvtk\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.938917 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.938890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-audit-log\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.939034 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.938951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.939514 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.939492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-metrics-server-audit-profiles\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.940994 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.940968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-client-ca-bundle\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.941100 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.941079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-secret-metrics-server-client-certs\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.941448 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.941422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-secret-metrics-server-tls\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:40.947052 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:40.947032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhvtk\" (UniqueName: \"kubernetes.io/projected/e5d7b414-8d85-4313-a7cd-52b7afb54ae8-kube-api-access-bhvtk\") pod \"metrics-server-6c7c4c7659-dn7nz\" (UID: \"e5d7b414-8d85-4313-a7cd-52b7afb54ae8\") " pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:41.079108 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:41.079083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:18:41.356540 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:41.356415    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"]
Sep 11 18:18:41.358957 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:18:41.358926    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d7b414_8d85_4313_a7cd_52b7afb54ae8.slice/crio-d33bde411d22fa5a902daeaa8327504ccffb580ff7744bfafdb1c86e573f5cf7 WatchSource:0}: Error finding container d33bde411d22fa5a902daeaa8327504ccffb580ff7744bfafdb1c86e573f5cf7: Status 404 returned error can't find the container with id d33bde411d22fa5a902daeaa8327504ccffb580ff7744bfafdb1c86e573f5cf7
Sep 11 18:18:41.603419 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:41.603387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz" event={"ID":"e5d7b414-8d85-4313-a7cd-52b7afb54ae8","Type":"ContainerStarted","Data":"d33bde411d22fa5a902daeaa8327504ccffb580ff7744bfafdb1c86e573f5cf7"}
Sep 11 18:18:41.605415 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:41.605388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p" event={"ID":"d5b7611c-a626-41d6-82f8-080815c87c27","Type":"ContainerStarted","Data":"267c85a880f7f7ce0281ce3bf2cdd82c3c9c772036a1e9b474ee133f6f1f5405"}
Sep 11 18:18:41.605515 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:41.605443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p" event={"ID":"d5b7611c-a626-41d6-82f8-080815c87c27","Type":"ContainerStarted","Data":"58f33f60b27a204ee39f063af94fd3f2ede37bc42db8a4b69ea17e40dd8edd6a"}
Sep 11 18:18:41.605515 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:41.605467    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p" event={"ID":"d5b7611c-a626-41d6-82f8-080815c87c27","Type":"ContainerStarted","Data":"e99759802c49f7f62d467d2eac93d8c5fbe93bd49d8e5b50333bbdd2e64eabd5"}
Sep 11 18:18:42.612834 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.612788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p" event={"ID":"d5b7611c-a626-41d6-82f8-080815c87c27","Type":"ContainerStarted","Data":"36c17ace0e8a9c580c2cc4b9018f0b5938971df9ca720109fd101dfea92af042"}
Sep 11 18:18:42.613238 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.612878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p" event={"ID":"d5b7611c-a626-41d6-82f8-080815c87c27","Type":"ContainerStarted","Data":"a2e24d9971ee2e8adcec7c3a8c7706d8aa9880c7713c0f866c68dda399fdfbfc"}
Sep 11 18:18:42.613238 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.612919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p" event={"ID":"d5b7611c-a626-41d6-82f8-080815c87c27","Type":"ContainerStarted","Data":"78946cc6ed086c9f2f64a41c6bdf2f342aa1d3f7149b69e8154db8e2fbe14b1c"}
Sep 11 18:18:42.613238 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.612982    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:42.638230 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.638087    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p" podStartSLOduration=1.647901217 podStartE2EDuration="4.638065901s" totalImagesPullingTime="2.990164684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:42.635083616 +0000 UTC m=+187.214348175" watchObservedRunningTime="2026-09-11 18:18:42.638065901 +0000 UTC m=+187.217330464"
Sep 11 18:18:42.736750 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.736716    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:18:42.741176 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.741126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.746192 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.746132    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 18:18:42.746192 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.746187    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-93vdidtihjcfc\""
Sep 11 18:18:42.746377 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.746138    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 18:18:42.746447 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.746426    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4dc7s\""
Sep 11 18:18:42.746552 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.746358    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 18:18:42.746685 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.746563    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 18:18:42.746743 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.746703    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 18:18:42.746743 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.746754    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 18:18:42.746857 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.746824    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 18:18:42.746980 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.746954    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 18:18:42.747060 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.747007    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 18:18:42.747120 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.747098    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 18:18:42.747188 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.747172    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 18:18:42.747316 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.747299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 18:18:42.747512 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.747495    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 18:18:42.748320 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.748295    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 18:18:42.758949 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.758453    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:18:42.853123 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853123 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853357 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853357 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853357 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853496 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853496 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96nt9\" (UniqueName: \"kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-kube-api-access-96nt9\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853496 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853623 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853623 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853623 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853623 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853794 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853794 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853794 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853794 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853955 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853955 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853955 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.853955 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.853920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.954605 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.954605 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.954605 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.954870 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.954870 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.954870 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.954870 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.954870 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.954870 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.954870 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.954979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.955003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.955018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.955038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.955072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96nt9\" (UniqueName: \"kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-kube-api-access-96nt9\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.955094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.955125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.955151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.955184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.955173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.956263 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.955891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.957407 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.957099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.957407 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.957306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.958982 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.958074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.958982 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.958613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.958982 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.958919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.959737 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.959713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.960164 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.960137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.960351 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.960328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.960706 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.960682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.960895 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.960872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.961519 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.961494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.962210 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.961923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.962354 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.962303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.962456 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.962388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.962780 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.962762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.962856 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.962768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.962907 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.962891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:42.969281 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:42.969256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96nt9\" (UniqueName: \"kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-kube-api-access-96nt9\") pod \"prometheus-k8s-0\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:43.055617 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:43.055592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:43.344295 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:43.344265    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:18:43.347081 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:18:43.347055    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692f8ce7_8f1a_4093_aeaf_90ba9b9595d2.slice/crio-db58c929172d3a71964983b4214a46d3d50f6416d1fed0af9f9094f1ce352a68 WatchSource:0}: Error finding container db58c929172d3a71964983b4214a46d3d50f6416d1fed0af9f9094f1ce352a68: Status 404 returned error can't find the container with id db58c929172d3a71964983b4214a46d3d50f6416d1fed0af9f9094f1ce352a68
Sep 11 18:18:43.618182 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:43.618095    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz" event={"ID":"e5d7b414-8d85-4313-a7cd-52b7afb54ae8","Type":"ContainerStarted","Data":"34ea9442d1f10a974cc9dcdf477e8dea8b6af6c12b7d9b4e82eb2a2b60ea3f8d"}
Sep 11 18:18:43.619285 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:43.619258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerStarted","Data":"db58c929172d3a71964983b4214a46d3d50f6416d1fed0af9f9094f1ce352a68"}
Sep 11 18:18:43.638667 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:43.638024    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz" podStartSLOduration=2.003042419 podStartE2EDuration="3.638005516s" totalImagesPullingTime="1.634963097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:43.63582946 +0000 UTC m=+188.215094212" watchObservedRunningTime="2026-09-11 18:18:43.638005516 +0000 UTC m=+188.217270076"
Sep 11 18:18:44.626970 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:44.626932    2578 generic.go:356] "Generic (PLEG): container finished" podID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerID="f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f" exitCode=0
Sep 11 18:18:44.627398 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:44.627017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerDied","Data":"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"}
Sep 11 18:18:48.626716 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:48.626605    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-588b5db496-jrr8p"
Sep 11 18:18:48.653733 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:48.653694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerStarted","Data":"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"}
Sep 11 18:18:48.653899 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:48.653821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerStarted","Data":"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"}
Sep 11 18:18:48.653899 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:48.653864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerStarted","Data":"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"}
Sep 11 18:18:48.653992 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:48.653907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerStarted","Data":"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"}
Sep 11 18:18:48.653992 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:48.653949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerStarted","Data":"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"}
Sep 11 18:18:48.653992 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:48.653984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerStarted","Data":"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"}
Sep 11 18:18:48.692608 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:48.692548    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.309837512 podStartE2EDuration="6.692295174s" totalImagesPullingTime="4.363562778s" totalInitContainerRuntime="18.894884ms" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:48.689959599 +0000 UTC m=+193.269224167" watchObservedRunningTime="2026-09-11 18:18:48.692295174 +0000 UTC m=+193.271559734"
Sep 11 18:18:53.056277 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:18:53.056246    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:19:01.079735 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:01.079694    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:19:01.080106 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:01.079757    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:19:21.085204 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:21.085168    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:19:21.088948 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:21.088926    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6c7c4c7659-dn7nz"
Sep 11 18:19:43.055867 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:43.055830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:19:43.074162 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:43.074135    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:19:43.866831 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:43.866807    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:19:47.744595 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:47.744560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:19:47.746902 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:47.746877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c498ff-9632-4d5a-8dad-d1d9a94137e1-metrics-certs\") pod \"network-metrics-daemon-sv62g\" (UID: \"78c498ff-9632-4d5a-8dad-d1d9a94137e1\") " pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:19:47.935117 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:47.935091    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kq7d5\""
Sep 11 18:19:47.942392 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:47.942369    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv62g"
Sep 11 18:19:48.209067 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:48.209031    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sv62g"]
Sep 11 18:19:48.211956 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:19:48.211931    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c498ff_9632_4d5a_8dad_d1d9a94137e1.slice/crio-fbe5293626aa85fdd19c6b0f37d474b3b913fd1a7ae406dc0e47db38e1a15f50 WatchSource:0}: Error finding container fbe5293626aa85fdd19c6b0f37d474b3b913fd1a7ae406dc0e47db38e1a15f50: Status 404 returned error can't find the container with id fbe5293626aa85fdd19c6b0f37d474b3b913fd1a7ae406dc0e47db38e1a15f50
Sep 11 18:19:48.875132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:48.875090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sv62g" event={"ID":"78c498ff-9632-4d5a-8dad-d1d9a94137e1","Type":"ContainerStarted","Data":"fbe5293626aa85fdd19c6b0f37d474b3b913fd1a7ae406dc0e47db38e1a15f50"}
Sep 11 18:19:49.880773 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:49.880733    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sv62g" event={"ID":"78c498ff-9632-4d5a-8dad-d1d9a94137e1","Type":"ContainerStarted","Data":"6d2213e2e912b6733e856b784405dd70a3d1bd22957dc30d3685f7370754ca28"}
Sep 11 18:19:49.881156 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:49.880795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sv62g" event={"ID":"78c498ff-9632-4d5a-8dad-d1d9a94137e1","Type":"ContainerStarted","Data":"9a0c1b4955f98a0bbc782fa1bb98a370895d123807c2df8f516c9a2084d5beda"}
Sep 11 18:19:49.901903 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:19:49.901859    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sv62g" podStartSLOduration=253.130275745 podStartE2EDuration="4m13.901843431s" totalImagesPullingTime="771.567686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:49.900013759 +0000 UTC m=+254.479278329" watchObservedRunningTime="2026-09-11 18:19:49.901843431 +0000 UTC m=+254.481107991"
Sep 11 18:20:01.104027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.103946    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:01.104508 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.104234    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="prometheus" containerID="cri-o://bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6" gracePeriod=600
Sep 11 18:20:01.104508 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.104247    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy" containerID="cri-o://9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc" gracePeriod=600
Sep 11 18:20:01.104508 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.104311    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="thanos-sidecar" containerID="cri-o://8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95" gracePeriod=600
Sep 11 18:20:01.104508 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.104319    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="config-reloader" containerID="cri-o://21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03" gracePeriod=600
Sep 11 18:20:01.104508 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.104282    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy-web" containerID="cri-o://53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568" gracePeriod=600
Sep 11 18:20:01.104508 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.104264    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372" gracePeriod=600
Sep 11 18:20:01.363699 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.363623    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:01.447895 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.447869    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-0\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.447895 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.447900    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-tls\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.447924    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-thanos-prometheus-http-client-file\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.447943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-1\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.447970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config-out\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.447992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-grpc-tls\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448017    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-tls-assets\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448042    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-web-config\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448074    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-kubelet-serving-ca-bundle\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448448 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:01.448448 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-db\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448448 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448384    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-2\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448448 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96nt9\" (UniqueName: \"kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-kube-api-access-96nt9\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448639 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:01.448639 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-kube-rbac-proxy\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448639 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448523    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448639 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448554    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-serving-certs-ca-bundle\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448639 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-trusted-ca-bundle\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448639 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448610    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448639 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448992 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448696    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-metrics-client-certs\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448992 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-metrics-client-ca\") pod \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\" (UID: \"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2\") "
Sep 11 18:20:01.448992 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.448826    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:01.449143 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.449040    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.449143 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.449061    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.449143 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.449075    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.449143 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.449097    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:01.450047 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.449944    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:20:01.451494 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.450901    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:01.451494 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.451248    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:01.451494 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.451477    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:01.451736 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.451680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:01.452090 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.452064    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-kube-api-access-96nt9" (OuterVolumeSpecName: "kube-api-access-96nt9") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "kube-api-access-96nt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:20:01.452190 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.452088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:01.452190 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.452123    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:01.452729 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.452693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:20:01.452830 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.452805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config-out" (OuterVolumeSpecName: "config-out") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:20:01.454240 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.454213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:01.454240 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.454232    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:01.454404 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.454263    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:01.454470 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.454400    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config" (OuterVolumeSpecName: "config") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:01.454526 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.454469    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:01.462134 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.462115    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-web-config" (OuterVolumeSpecName: "web-config") pod "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" (UID: "692f8ce7-8f1a-4093-aeaf-90ba9b9595d2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:20:01.550399 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550376    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-db\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550399 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550397    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96nt9\" (UniqueName: \"kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-kube-api-access-96nt9\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550407    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-kube-rbac-proxy\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550417    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550426    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550434    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550443    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550451    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550460    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-metrics-client-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550467    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-configmap-metrics-client-ca\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550475    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550483    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550490    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550498    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-config-out\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550506    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-secret-grpc-tls\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550513    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-tls-assets\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.550525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.550521    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2-web-config\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:20:01.928027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.927994    2578 generic.go:356] "Generic (PLEG): container finished" podID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerID="62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372" exitCode=0
Sep 11 18:20:01.928027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928023    2578 generic.go:356] "Generic (PLEG): container finished" podID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerID="9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc" exitCode=0
Sep 11 18:20:01.928027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928033    2578 generic.go:356] "Generic (PLEG): container finished" podID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerID="53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568" exitCode=0
Sep 11 18:20:01.928236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928041    2578 generic.go:356] "Generic (PLEG): container finished" podID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerID="8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95" exitCode=0
Sep 11 18:20:01.928236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928049    2578 generic.go:356] "Generic (PLEG): container finished" podID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerID="21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03" exitCode=0
Sep 11 18:20:01.928236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928057    2578 generic.go:356] "Generic (PLEG): container finished" podID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerID="bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6" exitCode=0
Sep 11 18:20:01.928236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerDied","Data":"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"}
Sep 11 18:20:01.928236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerDied","Data":"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"}
Sep 11 18:20:01.928236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928082    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:01.928236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerDied","Data":"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"}
Sep 11 18:20:01.928236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerDied","Data":"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"}
Sep 11 18:20:01.928236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerDied","Data":"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"}
Sep 11 18:20:01.928545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928241    2578 scope.go:117] "RemoveContainer" containerID="62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"
Sep 11 18:20:01.928545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerDied","Data":"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"}
Sep 11 18:20:01.928545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.928270    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"692f8ce7-8f1a-4093-aeaf-90ba9b9595d2","Type":"ContainerDied","Data":"db58c929172d3a71964983b4214a46d3d50f6416d1fed0af9f9094f1ce352a68"}
Sep 11 18:20:01.936276 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.936137    2578 scope.go:117] "RemoveContainer" containerID="9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"
Sep 11 18:20:01.943335 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.943319    2578 scope.go:117] "RemoveContainer" containerID="53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"
Sep 11 18:20:01.951231 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.951162    2578 scope.go:117] "RemoveContainer" containerID="8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"
Sep 11 18:20:01.953609 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.953586    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:01.957161 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.957136    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:01.958751 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.958737    2578 scope.go:117] "RemoveContainer" containerID="21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"
Sep 11 18:20:01.964828 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.964812    2578 scope.go:117] "RemoveContainer" containerID="bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"
Sep 11 18:20:01.971436 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.971414    2578 scope.go:117] "RemoveContainer" containerID="f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"
Sep 11 18:20:01.977329 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.977315    2578 scope.go:117] "RemoveContainer" containerID="62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"
Sep 11 18:20:01.977574 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:20:01.977559    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": container with ID starting with 62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372 not found: ID does not exist" containerID="62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"
Sep 11 18:20:01.977616 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.977583    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"} err="failed to get container status \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": rpc error: code = NotFound desc = could not find container \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": container with ID starting with 62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372 not found: ID does not exist"
Sep 11 18:20:01.977616 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.977599    2578 scope.go:117] "RemoveContainer" containerID="9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"
Sep 11 18:20:01.977852 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:20:01.977836    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": container with ID starting with 9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc not found: ID does not exist" containerID="9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"
Sep 11 18:20:01.977897 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.977858    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"} err="failed to get container status \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": rpc error: code = NotFound desc = could not find container \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": container with ID starting with 9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc not found: ID does not exist"
Sep 11 18:20:01.977897 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.977878    2578 scope.go:117] "RemoveContainer" containerID="53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"
Sep 11 18:20:01.978106 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:20:01.978088    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": container with ID starting with 53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568 not found: ID does not exist" containerID="53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"
Sep 11 18:20:01.978165 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.978116    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"} err="failed to get container status \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": rpc error: code = NotFound desc = could not find container \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": container with ID starting with 53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568 not found: ID does not exist"
Sep 11 18:20:01.978165 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.978137    2578 scope.go:117] "RemoveContainer" containerID="8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"
Sep 11 18:20:01.978377 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:20:01.978358    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": container with ID starting with 8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95 not found: ID does not exist" containerID="8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"
Sep 11 18:20:01.978419 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.978396    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"} err="failed to get container status \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": rpc error: code = NotFound desc = could not find container \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": container with ID starting with 8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95 not found: ID does not exist"
Sep 11 18:20:01.978419 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.978409    2578 scope.go:117] "RemoveContainer" containerID="21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"
Sep 11 18:20:01.978624 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:20:01.978607    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": container with ID starting with 21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03 not found: ID does not exist" containerID="21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"
Sep 11 18:20:01.978681 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.978628    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"} err="failed to get container status \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": rpc error: code = NotFound desc = could not find container \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": container with ID starting with 21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03 not found: ID does not exist"
Sep 11 18:20:01.978737 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.978642    2578 scope.go:117] "RemoveContainer" containerID="bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"
Sep 11 18:20:01.978969 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:20:01.978954    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": container with ID starting with bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6 not found: ID does not exist" containerID="bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"
Sep 11 18:20:01.979031 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.978974    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"} err="failed to get container status \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": rpc error: code = NotFound desc = could not find container \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": container with ID starting with bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6 not found: ID does not exist"
Sep 11 18:20:01.979031 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.978993    2578 scope.go:117] "RemoveContainer" containerID="f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"
Sep 11 18:20:01.979234 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:20:01.979219    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": container with ID starting with f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f not found: ID does not exist" containerID="f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"
Sep 11 18:20:01.979280 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.979237    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"} err="failed to get container status \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": rpc error: code = NotFound desc = could not find container \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": container with ID starting with f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f not found: ID does not exist"
Sep 11 18:20:01.979280 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.979251    2578 scope.go:117] "RemoveContainer" containerID="62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"
Sep 11 18:20:01.979452 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.979431    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"} err="failed to get container status \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": rpc error: code = NotFound desc = could not find container \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": container with ID starting with 62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372 not found: ID does not exist"
Sep 11 18:20:01.979488 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.979453    2578 scope.go:117] "RemoveContainer" containerID="9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"
Sep 11 18:20:01.979672 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.979642    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"} err="failed to get container status \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": rpc error: code = NotFound desc = could not find container \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": container with ID starting with 9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc not found: ID does not exist"
Sep 11 18:20:01.979709 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.979673    2578 scope.go:117] "RemoveContainer" containerID="53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"
Sep 11 18:20:01.979908 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.979890    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"} err="failed to get container status \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": rpc error: code = NotFound desc = could not find container \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": container with ID starting with 53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568 not found: ID does not exist"
Sep 11 18:20:01.979958 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.979909    2578 scope.go:117] "RemoveContainer" containerID="8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"
Sep 11 18:20:01.980141 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.980124    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"} err="failed to get container status \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": rpc error: code = NotFound desc = could not find container \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": container with ID starting with 8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95 not found: ID does not exist"
Sep 11 18:20:01.980205 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.980143    2578 scope.go:117] "RemoveContainer" containerID="21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"
Sep 11 18:20:01.980375 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.980358    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"} err="failed to get container status \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": rpc error: code = NotFound desc = could not find container \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": container with ID starting with 21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03 not found: ID does not exist"
Sep 11 18:20:01.980442 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.980376    2578 scope.go:117] "RemoveContainer" containerID="bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"
Sep 11 18:20:01.980584 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.980569    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"} err="failed to get container status \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": rpc error: code = NotFound desc = could not find container \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": container with ID starting with bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6 not found: ID does not exist"
Sep 11 18:20:01.980698 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.980585    2578 scope.go:117] "RemoveContainer" containerID="f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"
Sep 11 18:20:01.980826 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.980810    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"} err="failed to get container status \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": rpc error: code = NotFound desc = could not find container \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": container with ID starting with f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f not found: ID does not exist"
Sep 11 18:20:01.980863 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.980827    2578 scope.go:117] "RemoveContainer" containerID="62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"
Sep 11 18:20:01.981082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.981059    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"} err="failed to get container status \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": rpc error: code = NotFound desc = could not find container \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": container with ID starting with 62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372 not found: ID does not exist"
Sep 11 18:20:01.981082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.981079    2578 scope.go:117] "RemoveContainer" containerID="9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"
Sep 11 18:20:01.981328 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.981303    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"} err="failed to get container status \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": rpc error: code = NotFound desc = could not find container \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": container with ID starting with 9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc not found: ID does not exist"
Sep 11 18:20:01.981426 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.981328    2578 scope.go:117] "RemoveContainer" containerID="53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"
Sep 11 18:20:01.981710 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.981689    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"} err="failed to get container status \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": rpc error: code = NotFound desc = could not find container \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": container with ID starting with 53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568 not found: ID does not exist"
Sep 11 18:20:01.981801 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.981711    2578 scope.go:117] "RemoveContainer" containerID="8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"
Sep 11 18:20:01.982099 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.982071    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"} err="failed to get container status \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": rpc error: code = NotFound desc = could not find container \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": container with ID starting with 8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95 not found: ID does not exist"
Sep 11 18:20:01.982099 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.982098    2578 scope.go:117] "RemoveContainer" containerID="21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"
Sep 11 18:20:01.982375 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.982357    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"} err="failed to get container status \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": rpc error: code = NotFound desc = could not find container \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": container with ID starting with 21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03 not found: ID does not exist"
Sep 11 18:20:01.982375 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.982375    2578 scope.go:117] "RemoveContainer" containerID="bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"
Sep 11 18:20:01.982640 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.982622    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"} err="failed to get container status \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": rpc error: code = NotFound desc = could not find container \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": container with ID starting with bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6 not found: ID does not exist"
Sep 11 18:20:01.982710 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.982640    2578 scope.go:117] "RemoveContainer" containerID="f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"
Sep 11 18:20:01.982886 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.982870    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"} err="failed to get container status \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": rpc error: code = NotFound desc = could not find container \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": container with ID starting with f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f not found: ID does not exist"
Sep 11 18:20:01.982886 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.982885    2578 scope.go:117] "RemoveContainer" containerID="62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"
Sep 11 18:20:01.983098 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983078    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"} err="failed to get container status \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": rpc error: code = NotFound desc = could not find container \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": container with ID starting with 62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372 not found: ID does not exist"
Sep 11 18:20:01.983138 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983098    2578 scope.go:117] "RemoveContainer" containerID="9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"
Sep 11 18:20:01.983290 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983275    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:01.983334 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983296    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"} err="failed to get container status \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": rpc error: code = NotFound desc = could not find container \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": container with ID starting with 9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc not found: ID does not exist"
Sep 11 18:20:01.983334 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983313    2578 scope.go:117] "RemoveContainer" containerID="53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"
Sep 11 18:20:01.983545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983525    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"} err="failed to get container status \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": rpc error: code = NotFound desc = could not find container \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": container with ID starting with 53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568 not found: ID does not exist"
Sep 11 18:20:01.983610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983551    2578 scope.go:117] "RemoveContainer" containerID="8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"
Sep 11 18:20:01.983610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983557    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="thanos-sidecar"
Sep 11 18:20:01.983610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983569    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="thanos-sidecar"
Sep 11 18:20:01.983610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983577    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="init-config-reloader"
Sep 11 18:20:01.983610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983582    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="init-config-reloader"
Sep 11 18:20:01.983610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983590    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="prometheus"
Sep 11 18:20:01.983610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983595    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="prometheus"
Sep 11 18:20:01.983610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983608    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="config-reloader"
Sep 11 18:20:01.983610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983613    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="config-reloader"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983620    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983626    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983634    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy-web"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983639    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy-web"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983644    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy-thanos"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983673    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy-thanos"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983734    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="config-reloader"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983743    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="thanos-sidecar"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983750    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy-web"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983755    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983760    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="prometheus"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983768    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" containerName="kube-rbac-proxy-thanos"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983815    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"} err="failed to get container status \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": rpc error: code = NotFound desc = could not find container \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": container with ID starting with 8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95 not found: ID does not exist"
Sep 11 18:20:01.983946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.983835    2578 scope.go:117] "RemoveContainer" containerID="21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"
Sep 11 18:20:01.984395 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.984078    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"} err="failed to get container status \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": rpc error: code = NotFound desc = could not find container \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": container with ID starting with 21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03 not found: ID does not exist"
Sep 11 18:20:01.984395 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.984092    2578 scope.go:117] "RemoveContainer" containerID="bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"
Sep 11 18:20:01.984395 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.984303    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"} err="failed to get container status \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": rpc error: code = NotFound desc = could not find container \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": container with ID starting with bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6 not found: ID does not exist"
Sep 11 18:20:01.984395 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.984318    2578 scope.go:117] "RemoveContainer" containerID="f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"
Sep 11 18:20:01.984551 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.984535    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"} err="failed to get container status \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": rpc error: code = NotFound desc = could not find container \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": container with ID starting with f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f not found: ID does not exist"
Sep 11 18:20:01.984592 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.984552    2578 scope.go:117] "RemoveContainer" containerID="62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"
Sep 11 18:20:01.984823 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.984806    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"} err="failed to get container status \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": rpc error: code = NotFound desc = could not find container \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": container with ID starting with 62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372 not found: ID does not exist"
Sep 11 18:20:01.984900 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.984825    2578 scope.go:117] "RemoveContainer" containerID="9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"
Sep 11 18:20:01.985062 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.985039    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"} err="failed to get container status \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": rpc error: code = NotFound desc = could not find container \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": container with ID starting with 9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc not found: ID does not exist"
Sep 11 18:20:01.985107 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.985062    2578 scope.go:117] "RemoveContainer" containerID="53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"
Sep 11 18:20:01.985277 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.985258    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"} err="failed to get container status \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": rpc error: code = NotFound desc = could not find container \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": container with ID starting with 53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568 not found: ID does not exist"
Sep 11 18:20:01.985341 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.985278    2578 scope.go:117] "RemoveContainer" containerID="8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"
Sep 11 18:20:01.985512 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.985497    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"} err="failed to get container status \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": rpc error: code = NotFound desc = could not find container \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": container with ID starting with 8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95 not found: ID does not exist"
Sep 11 18:20:01.985574 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.985513    2578 scope.go:117] "RemoveContainer" containerID="21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"
Sep 11 18:20:01.985725 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.985710    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"} err="failed to get container status \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": rpc error: code = NotFound desc = could not find container \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": container with ID starting with 21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03 not found: ID does not exist"
Sep 11 18:20:01.985795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.985727    2578 scope.go:117] "RemoveContainer" containerID="bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.985925    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"} err="failed to get container status \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": rpc error: code = NotFound desc = could not find container \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": container with ID starting with bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6 not found: ID does not exist"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.986233    2578 scope.go:117] "RemoveContainer" containerID="f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.986572    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"} err="failed to get container status \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": rpc error: code = NotFound desc = could not find container \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": container with ID starting with f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f not found: ID does not exist"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.986594    2578 scope.go:117] "RemoveContainer" containerID="62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.986879    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372"} err="failed to get container status \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": rpc error: code = NotFound desc = could not find container \"62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372\": container with ID starting with 62f218d5d746579025eb7af0b827ac9bea18fecf398863b6af4537dea7785372 not found: ID does not exist"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.986906    2578 scope.go:117] "RemoveContainer" containerID="9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.987165    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc"} err="failed to get container status \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": rpc error: code = NotFound desc = could not find container \"9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc\": container with ID starting with 9bf30008eb01705a499bed4cd1bc2bb337da3e814feeec09f9a75a1f1bf30fdc not found: ID does not exist"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.987191    2578 scope.go:117] "RemoveContainer" containerID="53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.987496    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568"} err="failed to get container status \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": rpc error: code = NotFound desc = could not find container \"53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568\": container with ID starting with 53916b3c8210c8ba8fee224dab5db863814c0bfb7e687f9b7ec9ed73f6358568 not found: ID does not exist"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.987516    2578 scope.go:117] "RemoveContainer" containerID="8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.988114    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95"} err="failed to get container status \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": rpc error: code = NotFound desc = could not find container \"8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95\": container with ID starting with 8a489a8bc211f8b0c0985e0a742a669522b6e2f7e2603e81b974586848314c95 not found: ID does not exist"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.988142    2578 scope.go:117] "RemoveContainer" containerID="21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.988405    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03"} err="failed to get container status \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": rpc error: code = NotFound desc = could not find container \"21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03\": container with ID starting with 21568e2177b7e4769055ebda8a889a1351bc956104e21a86d4c6e591ee42ee03 not found: ID does not exist"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.988429    2578 scope.go:117] "RemoveContainer" containerID="bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.989128    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6"} err="failed to get container status \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": rpc error: code = NotFound desc = could not find container \"bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6\": container with ID starting with bdde7fc34ea7941cd9c02e0435ca9af46fc0966d1ffce594ba5d962738f8d6a6 not found: ID does not exist"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.989148    2578 scope.go:117] "RemoveContainer" containerID="f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"
Sep 11 18:20:01.989545 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.989317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:01.990387 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.989876    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f"} err="failed to get container status \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": rpc error: code = NotFound desc = could not find container \"f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f\": container with ID starting with f9327643935a5c662068c85d59d100944460992bb0f8f56de50ecb464a1b311f not found: ID does not exist"
Sep 11 18:20:01.994339 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994315    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 18:20:01.994339 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994333    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 18:20:01.994479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994325    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 18:20:01.994479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994333    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 18:20:01.994479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994323    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 18:20:01.994636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994615    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4dc7s\""
Sep 11 18:20:01.994636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994631    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 18:20:01.994789 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994703    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 18:20:01.994789 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994717    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 18:20:01.994789 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994731    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-93vdidtihjcfc\""
Sep 11 18:20:01.994789 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994707    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 18:20:01.994789 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.994754    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 18:20:01.995171 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.995154    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 18:20:01.995243 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.995194    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 18:20:01.999044 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:01.999024    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 18:20:02.001527 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.001510    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 18:20:02.006105 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.006086    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:02.054858 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.054833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.054968 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.054861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70baba0f-ad60-47f2-b499-0c40808bf6a6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.054968 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.054878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.054968 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.054892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.054968 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.054913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.054968 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.054940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.054968 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.054955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.054972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-config\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055336 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055336 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97bf5\" (UniqueName: \"kubernetes.io/projected/70baba0f-ad60-47f2-b499-0c40808bf6a6-kube-api-access-97bf5\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055336 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70baba0f-ad60-47f2-b499-0c40808bf6a6-config-out\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055336 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055336 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-web-config\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.055336 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.055230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156235 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156506 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156506 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-config\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156506 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156506 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156506 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156506 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bf5\" (UniqueName: \"kubernetes.io/projected/70baba0f-ad60-47f2-b499-0c40808bf6a6-kube-api-access-97bf5\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70baba0f-ad60-47f2-b499-0c40808bf6a6-config-out\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-web-config\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70baba0f-ad60-47f2-b499-0c40808bf6a6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.156814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.157390 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.156846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.157390 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.157308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.157911 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.157889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.158163 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.158137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.159330 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.159286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-config\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.159330 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.159317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.159486 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.159331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.159896 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.159868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.159975 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.159921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.159975 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.159948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.160089 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.160068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70baba0f-ad60-47f2-b499-0c40808bf6a6-config-out\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.160563 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.160544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.160706 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.160604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70baba0f-ad60-47f2-b499-0c40808bf6a6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.160921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.160899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.162123 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.162099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-web-config\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.162244 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.162224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.162429 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.162414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.162731 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.162715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70baba0f-ad60-47f2-b499-0c40808bf6a6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.162936 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.162920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70baba0f-ad60-47f2-b499-0c40808bf6a6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.168064 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.168044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bf5\" (UniqueName: \"kubernetes.io/projected/70baba0f-ad60-47f2-b499-0c40808bf6a6-kube-api-access-97bf5\") pod \"prometheus-k8s-0\" (UID: \"70baba0f-ad60-47f2-b499-0c40808bf6a6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.301627 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.301593    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:02.588995 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.588926    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:20:02.593388 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:20:02.593359    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70baba0f_ad60_47f2_b499_0c40808bf6a6.slice/crio-af5768488013525fd5e5448723fbe71fc5c6982fc841c66854a9b9f284d65e81 WatchSource:0}: Error finding container af5768488013525fd5e5448723fbe71fc5c6982fc841c66854a9b9f284d65e81: Status 404 returned error can't find the container with id af5768488013525fd5e5448723fbe71fc5c6982fc841c66854a9b9f284d65e81
Sep 11 18:20:02.934239 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.934147    2578 generic.go:356] "Generic (PLEG): container finished" podID="70baba0f-ad60-47f2-b499-0c40808bf6a6" containerID="b6f2d1e5318b67f790b6150e3c36d1e21fa6a84648a13abe632bf667e9f94fe7" exitCode=0
Sep 11 18:20:02.934239 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.934235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70baba0f-ad60-47f2-b499-0c40808bf6a6","Type":"ContainerDied","Data":"b6f2d1e5318b67f790b6150e3c36d1e21fa6a84648a13abe632bf667e9f94fe7"}
Sep 11 18:20:02.934457 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:02.934301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70baba0f-ad60-47f2-b499-0c40808bf6a6","Type":"ContainerStarted","Data":"af5768488013525fd5e5448723fbe71fc5c6982fc841c66854a9b9f284d65e81"}
Sep 11 18:20:03.938017 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:03.937984    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692f8ce7-8f1a-4093-aeaf-90ba9b9595d2" path="/var/lib/kubelet/pods/692f8ce7-8f1a-4093-aeaf-90ba9b9595d2/volumes"
Sep 11 18:20:03.946678 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:03.946628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70baba0f-ad60-47f2-b499-0c40808bf6a6","Type":"ContainerStarted","Data":"bd97ddec5cb1bb6ff191d2791842bb0cfada047764d4fff36ab33d309237528d"}
Sep 11 18:20:03.946806 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:03.946752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70baba0f-ad60-47f2-b499-0c40808bf6a6","Type":"ContainerStarted","Data":"d0f6351b64fdb5b0a5363721cb7a2cbbc3a4638fc74a4b8a1d3b29dcde59b00f"}
Sep 11 18:20:03.946806 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:03.946793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70baba0f-ad60-47f2-b499-0c40808bf6a6","Type":"ContainerStarted","Data":"ec1570273809ab5d813680511f1491203e870d34adbb8c6873dfdbbb7bdea890"}
Sep 11 18:20:03.946867 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:03.946829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70baba0f-ad60-47f2-b499-0c40808bf6a6","Type":"ContainerStarted","Data":"917960cfc5bf22dedf598e138afb742cac60482d2c0985410a799778933c47c4"}
Sep 11 18:20:03.946896 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:03.946865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70baba0f-ad60-47f2-b499-0c40808bf6a6","Type":"ContainerStarted","Data":"55fbb3da7ea8c1b5d4b3e32a0f3c5222d55c406483ade611247338c6f99e4130"}
Sep 11 18:20:03.946928 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:03.946899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70baba0f-ad60-47f2-b499-0c40808bf6a6","Type":"ContainerStarted","Data":"d3a5d5866018f2240c99d5dfa68eb97183fbe97a3d08028ffef200af112acf81"}
Sep 11 18:20:03.980453 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:03.980400    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.960467649 podStartE2EDuration="2.980385021s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.917372ms" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:03.978777566 +0000 UTC m=+268.558042137" watchObservedRunningTime="2026-09-11 18:20:03.980385021 +0000 UTC m=+268.559649595"
Sep 11 18:20:07.302423 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:07.302388    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:20:35.854113 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:20:35.854070    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 18:21:02.301824 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:21:02.301791    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:21:02.316501 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:21:02.316477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:21:03.168256 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:21:03.168230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:23:19.096451 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.096413    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"]
Sep 11 18:23:19.099520 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.099505    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.102225 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.102201    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rqg22\""
Sep 11 18:23:19.103343 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.103326    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 18:23:19.103412 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.103330    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 18:23:19.103475 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.103459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 18:23:19.103574 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.103561    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:23:19.103678 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.103665    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 18:23:19.114325 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.114298    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"]
Sep 11 18:23:19.162659 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.162625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a843932-4f80-486d-81ec-a561a802469b-metrics-cert\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.162772 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.162743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a843932-4f80-486d-81ec-a561a802469b-cert\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.162836 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.162773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z58m8\" (UniqueName: \"kubernetes.io/projected/6a843932-4f80-486d-81ec-a561a802469b-kube-api-access-z58m8\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.162893 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.162834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a843932-4f80-486d-81ec-a561a802469b-manager-config\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.263695 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.263667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a843932-4f80-486d-81ec-a561a802469b-metrics-cert\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.263812 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.263716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a843932-4f80-486d-81ec-a561a802469b-cert\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.263812 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.263776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z58m8\" (UniqueName: \"kubernetes.io/projected/6a843932-4f80-486d-81ec-a561a802469b-kube-api-access-z58m8\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.263914 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.263820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a843932-4f80-486d-81ec-a561a802469b-manager-config\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.264400 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.264382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a843932-4f80-486d-81ec-a561a802469b-manager-config\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.266053 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.266036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a843932-4f80-486d-81ec-a561a802469b-metrics-cert\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.266115 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.266067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a843932-4f80-486d-81ec-a561a802469b-cert\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.281269 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.281247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z58m8\" (UniqueName: \"kubernetes.io/projected/6a843932-4f80-486d-81ec-a561a802469b-kube-api-access-z58m8\") pod \"lws-controller-manager-78b8f7c5cb-8ncpw\" (UID: \"6a843932-4f80-486d-81ec-a561a802469b\") " pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.408566 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.408518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:19.690260 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.690238    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"]
Sep 11 18:23:19.692448 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:23:19.692420    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a843932_4f80_486d_81ec_a561a802469b.slice/crio-41bfad4ec24cb7681b406d734ee9171c783e6acf8179502526cb0ab48628d177 WatchSource:0}: Error finding container 41bfad4ec24cb7681b406d734ee9171c783e6acf8179502526cb0ab48628d177: Status 404 returned error can't find the container with id 41bfad4ec24cb7681b406d734ee9171c783e6acf8179502526cb0ab48628d177
Sep 11 18:23:19.694297 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.694276    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:23:19.854978 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.854951    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"]
Sep 11 18:23:19.859232 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.859216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:19.863992 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.863973    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 18:23:19.864091 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.864005    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 18:23:19.864091 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.864033    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 18:23:19.864339 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.864323    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q6wtr\""
Sep 11 18:23:19.873072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.873057    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 18:23:19.884038 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.884016    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"]
Sep 11 18:23:19.970261 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.970209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:19.970353 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.970264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-manifests\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:19.970402 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.970348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-tmp\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:19.970402 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.970386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:19.970479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:19.970418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjwp\" (UniqueName: \"kubernetes.io/projected/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-kube-api-access-wtjwp\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.071752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.071726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-tmp\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.071856 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.071770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.071856 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.071814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjwp\" (UniqueName: \"kubernetes.io/projected/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-kube-api-access-wtjwp\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.071856 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.071841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.072005 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.071895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-manifests\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.072215 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.072196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-manifests\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.073851 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.073834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-tmp\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.074157 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.074141    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.074194 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.074176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.082397 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.082379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjwp\" (UniqueName: \"kubernetes.io/projected/d8a9b66c-2e36-405d-8d4a-78c24560f6fb-kube-api-access-wtjwp\") pod \"opendatahub-operator-controller-manager-68b7b6b756-bzkzq\" (UID: \"d8a9b66c-2e36-405d-8d4a-78c24560f6fb\") " pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.170000 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.169978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:20.475913 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.475852    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"]
Sep 11 18:23:20.481519 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:23:20.481451    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a9b66c_2e36_405d_8d4a_78c24560f6fb.slice/crio-b0f961ccddf00b13103d0c0b2a60890ea2ab659f1caecd8a0627bdac3cebf240 WatchSource:0}: Error finding container b0f961ccddf00b13103d0c0b2a60890ea2ab659f1caecd8a0627bdac3cebf240: Status 404 returned error can't find the container with id b0f961ccddf00b13103d0c0b2a60890ea2ab659f1caecd8a0627bdac3cebf240
Sep 11 18:23:20.649092 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.649056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq" event={"ID":"d8a9b66c-2e36-405d-8d4a-78c24560f6fb","Type":"ContainerStarted","Data":"b0f961ccddf00b13103d0c0b2a60890ea2ab659f1caecd8a0627bdac3cebf240"}
Sep 11 18:23:20.650562 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:20.650533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw" event={"ID":"6a843932-4f80-486d-81ec-a561a802469b","Type":"ContainerStarted","Data":"41bfad4ec24cb7681b406d734ee9171c783e6acf8179502526cb0ab48628d177"}
Sep 11 18:23:24.669347 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:24.669312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq" event={"ID":"d8a9b66c-2e36-405d-8d4a-78c24560f6fb","Type":"ContainerStarted","Data":"5bd357aff6dca3271bb358cd60c9f43c92609835ca7018a81eeef9a74cbd2ebf"}
Sep 11 18:23:24.670863 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:24.670840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw" event={"ID":"6a843932-4f80-486d-81ec-a561a802469b","Type":"ContainerStarted","Data":"2e513c99dc3527f8d5581657deb2cc355d7a8633cfdb5ab7631a81d5e37099a6"}
Sep 11 18:23:24.670967 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:24.670954    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:24.776316 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:24.776210    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw" podStartSLOduration=1.697193311 podStartE2EDuration="5.776192287s" totalImagesPullingTime="4.078998976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:24.775890394 +0000 UTC m=+469.355154949" watchObservedRunningTime="2026-09-11 18:23:24.776192287 +0000 UTC m=+469.355456846"
Sep 11 18:23:25.675595 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:25.675564    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8a9b66c-2e36-405d-8d4a-78c24560f6fb" containerID="5bd357aff6dca3271bb358cd60c9f43c92609835ca7018a81eeef9a74cbd2ebf" exitCode=0
Sep 11 18:23:25.675975 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:25.675641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq" event={"ID":"d8a9b66c-2e36-405d-8d4a-78c24560f6fb","Type":"ContainerDied","Data":"5bd357aff6dca3271bb358cd60c9f43c92609835ca7018a81eeef9a74cbd2ebf"}
Sep 11 18:23:26.680994 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:26.680958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq" event={"ID":"d8a9b66c-2e36-405d-8d4a-78c24560f6fb","Type":"ContainerStarted","Data":"d9c5f7bc87aecd92a90e0cfcce98b2d8d50c9c6358a5d9889de13520ce0d7d78"}
Sep 11 18:23:26.681334 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:26.681088    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:26.707934 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:26.707889    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq" podStartSLOduration=3.038618772 podStartE2EDuration="7.707876499s" totalImagesPullingTime="3.710932067s" totalInitContainerRuntime="958.32566ms" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:26.705166294 +0000 UTC m=+471.284430854" watchObservedRunningTime="2026-09-11 18:23:26.707876499 +0000 UTC m=+471.287141059"
Sep 11 18:23:35.678578 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:35.678543    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78b8f7c5cb-8ncpw"
Sep 11 18:23:37.687971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:37.687940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68b7b6b756-bzkzq"
Sep 11 18:23:39.898011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:39.897965    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"]
Sep 11 18:23:39.910861 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:39.910837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:39.916174 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:39.916139    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 18:23:39.917903 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:39.917428    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8c2dz\""
Sep 11 18:23:39.917903 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:39.917565    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 18:23:39.917903 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:39.917875    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 18:23:39.918141 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:39.918123    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 18:23:39.919045 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:39.919021    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"]
Sep 11 18:23:40.026980 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.026949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff434832-a415-4153-bd86-1cbace75a7a5-tls-certs\") pod \"kube-auth-proxy-5b58fb58cc-n4mp7\" (UID: \"ff434832-a415-4153-bd86-1cbace75a7a5\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:40.026980 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.026981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qdkj\" (UniqueName: \"kubernetes.io/projected/ff434832-a415-4153-bd86-1cbace75a7a5-kube-api-access-6qdkj\") pod \"kube-auth-proxy-5b58fb58cc-n4mp7\" (UID: \"ff434832-a415-4153-bd86-1cbace75a7a5\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:40.027159 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.027008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff434832-a415-4153-bd86-1cbace75a7a5-tmp\") pod \"kube-auth-proxy-5b58fb58cc-n4mp7\" (UID: \"ff434832-a415-4153-bd86-1cbace75a7a5\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:40.127612 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.127586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff434832-a415-4153-bd86-1cbace75a7a5-tls-certs\") pod \"kube-auth-proxy-5b58fb58cc-n4mp7\" (UID: \"ff434832-a415-4153-bd86-1cbace75a7a5\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:40.127816 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.127617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qdkj\" (UniqueName: \"kubernetes.io/projected/ff434832-a415-4153-bd86-1cbace75a7a5-kube-api-access-6qdkj\") pod \"kube-auth-proxy-5b58fb58cc-n4mp7\" (UID: \"ff434832-a415-4153-bd86-1cbace75a7a5\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:40.127816 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.127645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff434832-a415-4153-bd86-1cbace75a7a5-tmp\") pod \"kube-auth-proxy-5b58fb58cc-n4mp7\" (UID: \"ff434832-a415-4153-bd86-1cbace75a7a5\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:40.129741 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.129717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff434832-a415-4153-bd86-1cbace75a7a5-tmp\") pod \"kube-auth-proxy-5b58fb58cc-n4mp7\" (UID: \"ff434832-a415-4153-bd86-1cbace75a7a5\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:40.129980 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.129961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff434832-a415-4153-bd86-1cbace75a7a5-tls-certs\") pod \"kube-auth-proxy-5b58fb58cc-n4mp7\" (UID: \"ff434832-a415-4153-bd86-1cbace75a7a5\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:40.142320 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.142300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qdkj\" (UniqueName: \"kubernetes.io/projected/ff434832-a415-4153-bd86-1cbace75a7a5-kube-api-access-6qdkj\") pod \"kube-auth-proxy-5b58fb58cc-n4mp7\" (UID: \"ff434832-a415-4153-bd86-1cbace75a7a5\") " pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:40.221796 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.221737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"
Sep 11 18:23:40.497836 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.497810    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7"]
Sep 11 18:23:40.499680 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:23:40.499637    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff434832_a415_4153_bd86_1cbace75a7a5.slice/crio-5fe2fdd5036a0e6af42ef25bc4c3e3b19b8ac6bb97c3f4a03f818844bb948829 WatchSource:0}: Error finding container 5fe2fdd5036a0e6af42ef25bc4c3e3b19b8ac6bb97c3f4a03f818844bb948829: Status 404 returned error can't find the container with id 5fe2fdd5036a0e6af42ef25bc4c3e3b19b8ac6bb97c3f4a03f818844bb948829
Sep 11 18:23:40.735612 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:40.735580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7" event={"ID":"ff434832-a415-4153-bd86-1cbace75a7a5","Type":"ContainerStarted","Data":"5fe2fdd5036a0e6af42ef25bc4c3e3b19b8ac6bb97c3f4a03f818844bb948829"}
Sep 11 18:23:43.750369 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:43.750315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7" event={"ID":"ff434832-a415-4153-bd86-1cbace75a7a5","Type":"ContainerStarted","Data":"25c5f222188098d83062e9ac7d96560ef5ff2c2023b99f30213278e464ddad63"}
Sep 11 18:23:43.773017 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:23:43.772968    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b58fb58cc-n4mp7" podStartSLOduration=2.326502473 podStartE2EDuration="4.772953174s" totalImagesPullingTime="2.446450701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:43.771095113 +0000 UTC m=+488.350359674" watchObservedRunningTime="2026-09-11 18:23:43.772953174 +0000 UTC m=+488.352217734"
Sep 11 18:25:49.341220 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.341185    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"]
Sep 11 18:25:49.344377 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.344360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:49.347476 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.347455    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:25:49.347593 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.347522    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:25:49.347811 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.347791    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rzr4j\""
Sep 11 18:25:49.359488 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.359462    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"]
Sep 11 18:25:49.361304 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.361277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbdg\" (UniqueName: \"kubernetes.io/projected/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-kube-api-access-rhbdg\") pod \"kuadrant-operator-controller-manager-5f685c496f-pdm8m\" (UID: \"85d102d7-2ec7-45f7-b4ce-d38767db8ed0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:49.361396 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.361345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-pdm8m\" (UID: \"85d102d7-2ec7-45f7-b4ce-d38767db8ed0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:49.461814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.461792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbdg\" (UniqueName: \"kubernetes.io/projected/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-kube-api-access-rhbdg\") pod \"kuadrant-operator-controller-manager-5f685c496f-pdm8m\" (UID: \"85d102d7-2ec7-45f7-b4ce-d38767db8ed0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:49.461910 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.461826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-pdm8m\" (UID: \"85d102d7-2ec7-45f7-b4ce-d38767db8ed0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:49.462119 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.462103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-pdm8m\" (UID: \"85d102d7-2ec7-45f7-b4ce-d38767db8ed0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:49.476030 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.476012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbdg\" (UniqueName: \"kubernetes.io/projected/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-kube-api-access-rhbdg\") pod \"kuadrant-operator-controller-manager-5f685c496f-pdm8m\" (UID: \"85d102d7-2ec7-45f7-b4ce-d38767db8ed0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:49.653962 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.653908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:49.938942 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.938923    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"]
Sep 11 18:25:49.940863 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:25:49.940834    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d102d7_2ec7_45f7_b4ce_d38767db8ed0.slice/crio-6e48493464642ac19c938ddc6c5734d081f1009fe894b4eae0e94b04eedbebac WatchSource:0}: Error finding container 6e48493464642ac19c938ddc6c5734d081f1009fe894b4eae0e94b04eedbebac: Status 404 returned error can't find the container with id 6e48493464642ac19c938ddc6c5734d081f1009fe894b4eae0e94b04eedbebac
Sep 11 18:25:49.991738 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:49.991715    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"]
Sep 11 18:25:50.003322 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.003292    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"]
Sep 11 18:25:50.040538 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.040511    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"]
Sep 11 18:25:50.044747 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.044730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:25:50.067081 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.067057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-dlq67\" (UID: \"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:25:50.067178 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.067126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdjc\" (UniqueName: \"kubernetes.io/projected/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-kube-api-access-dfdjc\") pod \"kuadrant-operator-controller-manager-5f685c496f-dlq67\" (UID: \"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:25:50.096582 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.096541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"]
Sep 11 18:25:50.167851 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.167822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-dlq67\" (UID: \"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:25:50.167996 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.167893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdjc\" (UniqueName: \"kubernetes.io/projected/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-kube-api-access-dfdjc\") pod \"kuadrant-operator-controller-manager-5f685c496f-dlq67\" (UID: \"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:25:50.168219 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.168199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-dlq67\" (UID: \"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:25:50.179218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.179190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdjc\" (UniqueName: \"kubernetes.io/projected/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-kube-api-access-dfdjc\") pod \"kuadrant-operator-controller-manager-5f685c496f-dlq67\" (UID: \"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:25:50.354945 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.354834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:25:50.649110 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:50.649079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"]
Sep 11 18:25:50.650939 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:25:50.650911    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe8ae7f_d903_4a1a_92b4_6b37762b4e1c.slice/crio-c809bdaed20856da66da068f8ce2888e60b19c9fb119c9968c1cf485da4e1872 WatchSource:0}: Error finding container c809bdaed20856da66da068f8ce2888e60b19c9fb119c9968c1cf485da4e1872: Status 404 returned error can't find the container with id c809bdaed20856da66da068f8ce2888e60b19c9fb119c9968c1cf485da4e1872
Sep 11 18:25:51.254954 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:51.254917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67" event={"ID":"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c","Type":"ContainerStarted","Data":"c809bdaed20856da66da068f8ce2888e60b19c9fb119c9968c1cf485da4e1872"}
Sep 11 18:25:54.273857 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.273820    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m" podUID="85d102d7-2ec7-45f7-b4ce-d38767db8ed0" containerName="manager" containerID="cri-o://c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638" gracePeriod=2
Sep 11 18:25:54.275680 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.275635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67" event={"ID":"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c","Type":"ContainerStarted","Data":"300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49"}
Sep 11 18:25:54.275802 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.275772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:25:54.301904 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.301859    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67" podStartSLOduration=2.055440983 podStartE2EDuration="5.301844006s" totalImagesPullingTime="3.246403023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:54.299485773 +0000 UTC m=+618.878750344" watchObservedRunningTime="2026-09-11 18:25:54.301844006 +0000 UTC m=+618.881108567"
Sep 11 18:25:54.511212 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.511191    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:54.513941 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:25:54.513916    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-pdm8m\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" podUID="85d102d7-2ec7-45f7-b4ce-d38767db8ed0" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:54.598366 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.598309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-extensions-socket-volume\") pod \"85d102d7-2ec7-45f7-b4ce-d38767db8ed0\" (UID: \"85d102d7-2ec7-45f7-b4ce-d38767db8ed0\") "
Sep 11 18:25:54.598366 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.598358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhbdg\" (UniqueName: \"kubernetes.io/projected/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-kube-api-access-rhbdg\") pod \"85d102d7-2ec7-45f7-b4ce-d38767db8ed0\" (UID: \"85d102d7-2ec7-45f7-b4ce-d38767db8ed0\") "
Sep 11 18:25:54.598577 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.598556    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "85d102d7-2ec7-45f7-b4ce-d38767db8ed0" (UID: "85d102d7-2ec7-45f7-b4ce-d38767db8ed0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:25:54.600350 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.600328    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-kube-api-access-rhbdg" (OuterVolumeSpecName: "kube-api-access-rhbdg") pod "85d102d7-2ec7-45f7-b4ce-d38767db8ed0" (UID: "85d102d7-2ec7-45f7-b4ce-d38767db8ed0"). InnerVolumeSpecName "kube-api-access-rhbdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:25:54.698866 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.698842    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-extensions-socket-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:25:54.698866 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:54.698863    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhbdg\" (UniqueName: \"kubernetes.io/projected/85d102d7-2ec7-45f7-b4ce-d38767db8ed0-kube-api-access-rhbdg\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:25:55.280921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:55.280887    2578 generic.go:356] "Generic (PLEG): container finished" podID="85d102d7-2ec7-45f7-b4ce-d38767db8ed0" containerID="c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638" exitCode=2
Sep 11 18:25:55.281302 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:55.280932    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:55.281302 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:55.280987    2578 scope.go:117] "RemoveContainer" containerID="c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638"
Sep 11 18:25:55.283345 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:25:55.283315    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-pdm8m\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" podUID="85d102d7-2ec7-45f7-b4ce-d38767db8ed0" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:55.288769 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:55.288753    2578 scope.go:117] "RemoveContainer" containerID="c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638"
Sep 11 18:25:55.289019 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:25:55.289003    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638\": container with ID starting with c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638 not found: ID does not exist" containerID="c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638"
Sep 11 18:25:55.289086 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:55.289036    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638"} err="failed to get container status \"c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638\": rpc error: code = NotFound desc = could not find container \"c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638\": container with ID starting with c6280cedf9d09655d151f19b4aca9055721353a19b50c0bb9f0e831aee916638 not found: ID does not exist"
Sep 11 18:25:55.291206 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:25:55.291186    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-pdm8m\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" podUID="85d102d7-2ec7-45f7-b4ce-d38767db8ed0" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:55.935834 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:25:55.935799    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-pdm8m\" is forbidden: User \"system:node:ip-10-0-139-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-186.ec2.internal' and this object" podUID="85d102d7-2ec7-45f7-b4ce-d38767db8ed0" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pdm8m"
Sep 11 18:25:55.936757 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:25:55.936737    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85d102d7-2ec7-45f7-b4ce-d38767db8ed0" path="/var/lib/kubelet/pods/85d102d7-2ec7-45f7-b4ce-d38767db8ed0/volumes"
Sep 11 18:26:05.283395 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:05.283366    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:26:18.126745 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.126716    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"]
Sep 11 18:26:18.127108 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.126918    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67" podUID="6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c" containerName="manager" containerID="cri-o://300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49" gracePeriod=10
Sep 11 18:26:18.363193 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.363171    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:26:18.376352 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.376324    2578 generic.go:356] "Generic (PLEG): container finished" podID="6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c" containerID="300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49" exitCode=0
Sep 11 18:26:18.376443 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.376358    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"
Sep 11 18:26:18.376443 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.376387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67" event={"ID":"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c","Type":"ContainerDied","Data":"300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49"}
Sep 11 18:26:18.376443 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.376439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67" event={"ID":"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c","Type":"ContainerDied","Data":"c809bdaed20856da66da068f8ce2888e60b19c9fb119c9968c1cf485da4e1872"}
Sep 11 18:26:18.376606 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.376479    2578 scope.go:117] "RemoveContainer" containerID="300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49"
Sep 11 18:26:18.384804 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.384768    2578 scope.go:117] "RemoveContainer" containerID="300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49"
Sep 11 18:26:18.385099 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:26:18.385073    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49\": container with ID starting with 300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49 not found: ID does not exist" containerID="300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49"
Sep 11 18:26:18.385198 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.385110    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49"} err="failed to get container status \"300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49\": rpc error: code = NotFound desc = could not find container \"300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49\": container with ID starting with 300af25ad46050ea266af195b201ad5e523fc64ecbb4327a8683dcfbcb19ef49 not found: ID does not exist"
Sep 11 18:26:18.459535 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.459509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfdjc\" (UniqueName: \"kubernetes.io/projected/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-kube-api-access-dfdjc\") pod \"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c\" (UID: \"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c\") "
Sep 11 18:26:18.459636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.459559    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-extensions-socket-volume\") pod \"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c\" (UID: \"6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c\") "
Sep 11 18:26:18.460025 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.460006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c" (UID: "6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:26:18.461351 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.461332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-kube-api-access-dfdjc" (OuterVolumeSpecName: "kube-api-access-dfdjc") pod "6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c" (UID: "6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c"). InnerVolumeSpecName "kube-api-access-dfdjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:26:18.560863 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.560838    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfdjc\" (UniqueName: \"kubernetes.io/projected/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-kube-api-access-dfdjc\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:26:18.560863 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.560862    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c-extensions-socket-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:26:18.700367 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.700342    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"]
Sep 11 18:26:18.708275 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:18.708253    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-dlq67"]
Sep 11 18:26:19.936096 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:19.936049    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c" path="/var/lib/kubelet/pods/6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c/volumes"
Sep 11 18:26:38.613872 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.613841    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vcj26"]
Sep 11 18:26:38.614240 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.614146    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d102d7-2ec7-45f7-b4ce-d38767db8ed0" containerName="manager"
Sep 11 18:26:38.614240 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.614160    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d102d7-2ec7-45f7-b4ce-d38767db8ed0" containerName="manager"
Sep 11 18:26:38.614240 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.614169    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c" containerName="manager"
Sep 11 18:26:38.614240 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.614174    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c" containerName="manager"
Sep 11 18:26:38.614240 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.614238    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d102d7-2ec7-45f7-b4ce-d38767db8ed0" containerName="manager"
Sep 11 18:26:38.614401 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.614248    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe8ae7f-d903-4a1a-92b4-6b37762b4e1c" containerName="manager"
Sep 11 18:26:38.616909 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.616892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:38.621583 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.621555    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:26:38.621704 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.621596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:26:38.622690 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.622670    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 18:26:38.622690 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.622681    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g65q8\""
Sep 11 18:26:38.626726 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.626705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vcj26"]
Sep 11 18:26:38.699908 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.699879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-config-file\") pod \"limitador-limitador-8588749977-vcj26\" (UID: \"4fc2be97-3d0a-40f9-86a0-9c51972e3b24\") " pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:38.700039 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.700014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sss8w\" (UniqueName: \"kubernetes.io/projected/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-kube-api-access-sss8w\") pod \"limitador-limitador-8588749977-vcj26\" (UID: \"4fc2be97-3d0a-40f9-86a0-9c51972e3b24\") " pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:38.720439 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.720417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vcj26"]
Sep 11 18:26:38.801354 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.801323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-config-file\") pod \"limitador-limitador-8588749977-vcj26\" (UID: \"4fc2be97-3d0a-40f9-86a0-9c51972e3b24\") " pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:38.801482 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.801375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sss8w\" (UniqueName: \"kubernetes.io/projected/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-kube-api-access-sss8w\") pod \"limitador-limitador-8588749977-vcj26\" (UID: \"4fc2be97-3d0a-40f9-86a0-9c51972e3b24\") " pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:38.801902 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.801886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-config-file\") pod \"limitador-limitador-8588749977-vcj26\" (UID: \"4fc2be97-3d0a-40f9-86a0-9c51972e3b24\") " pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:38.811764 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.811744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sss8w\" (UniqueName: \"kubernetes.io/projected/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-kube-api-access-sss8w\") pod \"limitador-limitador-8588749977-vcj26\" (UID: \"4fc2be97-3d0a-40f9-86a0-9c51972e3b24\") " pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:38.928546 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:38.928491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:39.198336 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:39.198263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vcj26"]
Sep 11 18:26:39.200479 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:26:39.200452    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc2be97_3d0a_40f9_86a0_9c51972e3b24.slice/crio-65d271c42fc74f4630752a11955aec4db89e0ef9bb6653564fa28afb030f6a3f WatchSource:0}: Error finding container 65d271c42fc74f4630752a11955aec4db89e0ef9bb6653564fa28afb030f6a3f: Status 404 returned error can't find the container with id 65d271c42fc74f4630752a11955aec4db89e0ef9bb6653564fa28afb030f6a3f
Sep 11 18:26:39.433284 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:39.433255    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-js4f9"]
Sep 11 18:26:39.436008 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:39.435992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9"
Sep 11 18:26:39.438659 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:39.438627    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jgm7g\""
Sep 11 18:26:39.443986 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:39.443966    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-js4f9"]
Sep 11 18:26:39.468281 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:39.468230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-vcj26" event={"ID":"4fc2be97-3d0a-40f9-86a0-9c51972e3b24","Type":"ContainerStarted","Data":"65d271c42fc74f4630752a11955aec4db89e0ef9bb6653564fa28afb030f6a3f"}
Sep 11 18:26:39.506335 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:39.506301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9zg\" (UniqueName: \"kubernetes.io/projected/d25cc47c-50f0-4295-a879-c6982e9f71eb-kube-api-access-lt9zg\") pod \"authorino-7bcfd8f9cd-js4f9\" (UID: \"d25cc47c-50f0-4295-a879-c6982e9f71eb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9"
Sep 11 18:26:39.606822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:39.606789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9zg\" (UniqueName: \"kubernetes.io/projected/d25cc47c-50f0-4295-a879-c6982e9f71eb-kube-api-access-lt9zg\") pod \"authorino-7bcfd8f9cd-js4f9\" (UID: \"d25cc47c-50f0-4295-a879-c6982e9f71eb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9"
Sep 11 18:26:39.617951 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:39.617934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9zg\" (UniqueName: \"kubernetes.io/projected/d25cc47c-50f0-4295-a879-c6982e9f71eb-kube-api-access-lt9zg\") pod \"authorino-7bcfd8f9cd-js4f9\" (UID: \"d25cc47c-50f0-4295-a879-c6982e9f71eb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9"
Sep 11 18:26:39.746130 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:39.746071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9"
Sep 11 18:26:40.022802 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:40.022176    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-js4f9"]
Sep 11 18:26:40.473971 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:40.473930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-vcj26" event={"ID":"4fc2be97-3d0a-40f9-86a0-9c51972e3b24","Type":"ContainerStarted","Data":"958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d"}
Sep 11 18:26:40.474162 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:40.474028    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:40.475181 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:40.475157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9" event={"ID":"d25cc47c-50f0-4295-a879-c6982e9f71eb","Type":"ContainerStarted","Data":"45b21fb51f2138b592da779ab14d8c92566bfb10226e9d8e4f53808a783df31e"}
Sep 11 18:26:40.493130 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:40.493086    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-vcj26" podStartSLOduration=1.3811710879999999 podStartE2EDuration="2.493071676s" totalImagesPullingTime="1.111900588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:40.492100819 +0000 UTC m=+665.071365380" watchObservedRunningTime="2026-09-11 18:26:40.493071676 +0000 UTC m=+665.072336273"
Sep 11 18:26:42.324203 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:42.324174    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-js4f9"]
Sep 11 18:26:42.487384 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:42.487352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9" event={"ID":"d25cc47c-50f0-4295-a879-c6982e9f71eb","Type":"ContainerStarted","Data":"57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1"}
Sep 11 18:26:42.504557 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:42.504507    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9" podStartSLOduration=1.842371639 podStartE2EDuration="3.504493344s" totalImagesPullingTime="1.662121705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:42.503615598 +0000 UTC m=+667.082880157" watchObservedRunningTime="2026-09-11 18:26:42.504493344 +0000 UTC m=+667.083757903"
Sep 11 18:26:43.491176 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:43.491138    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9" podUID="d25cc47c-50f0-4295-a879-c6982e9f71eb" containerName="authorino" containerID="cri-o://57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1" gracePeriod=30
Sep 11 18:26:43.730413 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:43.730392    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9"
Sep 11 18:26:43.840221 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:43.840197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt9zg\" (UniqueName: \"kubernetes.io/projected/d25cc47c-50f0-4295-a879-c6982e9f71eb-kube-api-access-lt9zg\") pod \"d25cc47c-50f0-4295-a879-c6982e9f71eb\" (UID: \"d25cc47c-50f0-4295-a879-c6982e9f71eb\") "
Sep 11 18:26:43.842050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:43.842028    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25cc47c-50f0-4295-a879-c6982e9f71eb-kube-api-access-lt9zg" (OuterVolumeSpecName: "kube-api-access-lt9zg") pod "d25cc47c-50f0-4295-a879-c6982e9f71eb" (UID: "d25cc47c-50f0-4295-a879-c6982e9f71eb"). InnerVolumeSpecName "kube-api-access-lt9zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:26:43.940798 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:43.940777    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9zg\" (UniqueName: \"kubernetes.io/projected/d25cc47c-50f0-4295-a879-c6982e9f71eb-kube-api-access-lt9zg\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:26:44.496666 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:44.496624    2578 generic.go:356] "Generic (PLEG): container finished" podID="d25cc47c-50f0-4295-a879-c6982e9f71eb" containerID="57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1" exitCode=0
Sep 11 18:26:44.497043 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:44.496675    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9"
Sep 11 18:26:44.497043 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:44.496693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9" event={"ID":"d25cc47c-50f0-4295-a879-c6982e9f71eb","Type":"ContainerDied","Data":"57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1"}
Sep 11 18:26:44.497043 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:44.496726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-js4f9" event={"ID":"d25cc47c-50f0-4295-a879-c6982e9f71eb","Type":"ContainerDied","Data":"45b21fb51f2138b592da779ab14d8c92566bfb10226e9d8e4f53808a783df31e"}
Sep 11 18:26:44.497043 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:44.496741    2578 scope.go:117] "RemoveContainer" containerID="57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1"
Sep 11 18:26:44.504521 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:44.504503    2578 scope.go:117] "RemoveContainer" containerID="57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1"
Sep 11 18:26:44.504807 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:26:44.504791    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1\": container with ID starting with 57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1 not found: ID does not exist" containerID="57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1"
Sep 11 18:26:44.504861 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:44.504816    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1"} err="failed to get container status \"57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1\": rpc error: code = NotFound desc = could not find container \"57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1\": container with ID starting with 57312e46c9b6a92885f96cc9bfc623f11dc3a6ebd6ceab54562983413f8640e1 not found: ID does not exist"
Sep 11 18:26:44.516428 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:44.516405    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-js4f9"]
Sep 11 18:26:44.518240 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:44.518215    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-js4f9"]
Sep 11 18:26:45.936857 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:45.936825    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25cc47c-50f0-4295-a879-c6982e9f71eb" path="/var/lib/kubelet/pods/d25cc47c-50f0-4295-a879-c6982e9f71eb/volumes"
Sep 11 18:26:51.481703 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:51.481673    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:52.321173 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:52.321145    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vcj26"]
Sep 11 18:26:52.321423 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:52.321323    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-vcj26" podUID="4fc2be97-3d0a-40f9-86a0-9c51972e3b24" containerName="limitador" containerID="cri-o://958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d" gracePeriod=30
Sep 11 18:26:52.851537 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:52.851515    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:52.905810 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:52.905753    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-config-file\") pod \"4fc2be97-3d0a-40f9-86a0-9c51972e3b24\" (UID: \"4fc2be97-3d0a-40f9-86a0-9c51972e3b24\") "
Sep 11 18:26:52.905916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:52.905844    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sss8w\" (UniqueName: \"kubernetes.io/projected/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-kube-api-access-sss8w\") pod \"4fc2be97-3d0a-40f9-86a0-9c51972e3b24\" (UID: \"4fc2be97-3d0a-40f9-86a0-9c51972e3b24\") "
Sep 11 18:26:52.906073 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:52.906049    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-config-file" (OuterVolumeSpecName: "config-file") pod "4fc2be97-3d0a-40f9-86a0-9c51972e3b24" (UID: "4fc2be97-3d0a-40f9-86a0-9c51972e3b24"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:26:52.907733 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:52.907711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-kube-api-access-sss8w" (OuterVolumeSpecName: "kube-api-access-sss8w") pod "4fc2be97-3d0a-40f9-86a0-9c51972e3b24" (UID: "4fc2be97-3d0a-40f9-86a0-9c51972e3b24"). InnerVolumeSpecName "kube-api-access-sss8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:26:53.006401 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.006374    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sss8w\" (UniqueName: \"kubernetes.io/projected/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-kube-api-access-sss8w\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:26:53.006401 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.006400    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4fc2be97-3d0a-40f9-86a0-9c51972e3b24-config-file\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:26:53.534509 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.534471    2578 generic.go:356] "Generic (PLEG): container finished" podID="4fc2be97-3d0a-40f9-86a0-9c51972e3b24" containerID="958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d" exitCode=0
Sep 11 18:26:53.534666 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.534513    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-vcj26"
Sep 11 18:26:53.534666 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.534553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-vcj26" event={"ID":"4fc2be97-3d0a-40f9-86a0-9c51972e3b24","Type":"ContainerDied","Data":"958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d"}
Sep 11 18:26:53.534666 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.534603    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-vcj26" event={"ID":"4fc2be97-3d0a-40f9-86a0-9c51972e3b24","Type":"ContainerDied","Data":"65d271c42fc74f4630752a11955aec4db89e0ef9bb6653564fa28afb030f6a3f"}
Sep 11 18:26:53.534666 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.534627    2578 scope.go:117] "RemoveContainer" containerID="958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d"
Sep 11 18:26:53.542579 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.542563    2578 scope.go:117] "RemoveContainer" containerID="958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d"
Sep 11 18:26:53.542843 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:26:53.542823    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d\": container with ID starting with 958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d not found: ID does not exist" containerID="958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d"
Sep 11 18:26:53.542889 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.542853    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d"} err="failed to get container status \"958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d\": rpc error: code = NotFound desc = could not find container \"958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d\": container with ID starting with 958ad6b9fb2932914b3d0330303bae744e6d49772c0da616277ee79a9c4ccf7d not found: ID does not exist"
Sep 11 18:26:53.560488 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.560459    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vcj26"]
Sep 11 18:26:53.565095 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.565075    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vcj26"]
Sep 11 18:26:53.936493 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:53.936424    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc2be97-3d0a-40f9-86a0-9c51972e3b24" path="/var/lib/kubelet/pods/4fc2be97-3d0a-40f9-86a0-9c51972e3b24/volumes"
Sep 11 18:26:55.982946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.982918    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"]
Sep 11 18:26:55.983314 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.983199    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc2be97-3d0a-40f9-86a0-9c51972e3b24" containerName="limitador"
Sep 11 18:26:55.983314 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.983209    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc2be97-3d0a-40f9-86a0-9c51972e3b24" containerName="limitador"
Sep 11 18:26:55.983314 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.983221    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25cc47c-50f0-4295-a879-c6982e9f71eb" containerName="authorino"
Sep 11 18:26:55.983314 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.983226    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25cc47c-50f0-4295-a879-c6982e9f71eb" containerName="authorino"
Sep 11 18:26:55.983314 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.983276    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc2be97-3d0a-40f9-86a0-9c51972e3b24" containerName="limitador"
Sep 11 18:26:55.983314 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.983284    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25cc47c-50f0-4295-a879-c6982e9f71eb" containerName="authorino"
Sep 11 18:26:55.985694 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.985678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"
Sep 11 18:26:55.988621 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.988602    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nmh8n\""
Sep 11 18:26:55.988911 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.988893    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:26:55.989840 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.989823    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:26:55.989921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.989868    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 18:26:55.996831 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:55.996808    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"]
Sep 11 18:26:56.130768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:56.130745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hckj7\" (UniqueName: \"kubernetes.io/projected/9f4b16a2-88d3-4687-b55a-9fe04887351c-kube-api-access-hckj7\") pod \"postgres-69fb4594cf-vlmmq\" (UID: \"9f4b16a2-88d3-4687-b55a-9fe04887351c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"
Sep 11 18:26:56.130901 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:56.130794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9f4b16a2-88d3-4687-b55a-9fe04887351c-data\") pod \"postgres-69fb4594cf-vlmmq\" (UID: \"9f4b16a2-88d3-4687-b55a-9fe04887351c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"
Sep 11 18:26:56.232062 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:56.232038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hckj7\" (UniqueName: \"kubernetes.io/projected/9f4b16a2-88d3-4687-b55a-9fe04887351c-kube-api-access-hckj7\") pod \"postgres-69fb4594cf-vlmmq\" (UID: \"9f4b16a2-88d3-4687-b55a-9fe04887351c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"
Sep 11 18:26:56.232168 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:56.232078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9f4b16a2-88d3-4687-b55a-9fe04887351c-data\") pod \"postgres-69fb4594cf-vlmmq\" (UID: \"9f4b16a2-88d3-4687-b55a-9fe04887351c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"
Sep 11 18:26:56.232386 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:56.232373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9f4b16a2-88d3-4687-b55a-9fe04887351c-data\") pod \"postgres-69fb4594cf-vlmmq\" (UID: \"9f4b16a2-88d3-4687-b55a-9fe04887351c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"
Sep 11 18:26:56.241270 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:56.241210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hckj7\" (UniqueName: \"kubernetes.io/projected/9f4b16a2-88d3-4687-b55a-9fe04887351c-kube-api-access-hckj7\") pod \"postgres-69fb4594cf-vlmmq\" (UID: \"9f4b16a2-88d3-4687-b55a-9fe04887351c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"
Sep 11 18:26:56.296304 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:56.296283    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"
Sep 11 18:26:56.565102 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:56.564990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"]
Sep 11 18:26:56.567730 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:26:56.567699    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4b16a2_88d3_4687_b55a_9fe04887351c.slice/crio-de5cec733c295e4ebddd17cce02063b5c2cec7fcdc8a99ac43d3f77cc872d3e4 WatchSource:0}: Error finding container de5cec733c295e4ebddd17cce02063b5c2cec7fcdc8a99ac43d3f77cc872d3e4: Status 404 returned error can't find the container with id de5cec733c295e4ebddd17cce02063b5c2cec7fcdc8a99ac43d3f77cc872d3e4
Sep 11 18:26:57.554185 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:26:57.554147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq" event={"ID":"9f4b16a2-88d3-4687-b55a-9fe04887351c","Type":"ContainerStarted","Data":"de5cec733c295e4ebddd17cce02063b5c2cec7fcdc8a99ac43d3f77cc872d3e4"}
Sep 11 18:27:01.576488 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:01.576453    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq" event={"ID":"9f4b16a2-88d3-4687-b55a-9fe04887351c","Type":"ContainerStarted","Data":"9bd2c1d032a3fa11c69a9a662a5e5994622ed25cf95ad50c0f14cd63bc54219b"}
Sep 11 18:27:01.576958 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:01.576530    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"
Sep 11 18:27:01.595859 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:01.595818    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq" podStartSLOduration=2.007572421 podStartE2EDuration="6.595803169s" totalImagesPullingTime="4.588230748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:01.595631139 +0000 UTC m=+686.174895698" watchObservedRunningTime="2026-09-11 18:27:01.595803169 +0000 UTC m=+686.175067725"
Sep 11 18:27:07.610403 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:07.610372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlmmq"
Sep 11 18:27:08.138960 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.138931    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-xwx5h"]
Sep 11 18:27:08.144559 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.144541    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xwx5h"
Sep 11 18:27:08.148485 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.148454    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:27:08.149537 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.149512    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jgm7g\""
Sep 11 18:27:08.149683 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.149665    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:27:08.151287 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.151267    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-xwx5h"]
Sep 11 18:27:08.226023 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.226000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nznfm\" (UniqueName: \"kubernetes.io/projected/dadd228d-09fb-4e5c-be66-13cdb74b60b0-kube-api-access-nznfm\") pod \"authorino-557f876f69-xwx5h\" (UID: \"dadd228d-09fb-4e5c-be66-13cdb74b60b0\") " pod="kuadrant-system/authorino-557f876f69-xwx5h"
Sep 11 18:27:08.326899 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.326877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nznfm\" (UniqueName: \"kubernetes.io/projected/dadd228d-09fb-4e5c-be66-13cdb74b60b0-kube-api-access-nznfm\") pod \"authorino-557f876f69-xwx5h\" (UID: \"dadd228d-09fb-4e5c-be66-13cdb74b60b0\") " pod="kuadrant-system/authorino-557f876f69-xwx5h"
Sep 11 18:27:08.335687 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.335645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nznfm\" (UniqueName: \"kubernetes.io/projected/dadd228d-09fb-4e5c-be66-13cdb74b60b0-kube-api-access-nznfm\") pod \"authorino-557f876f69-xwx5h\" (UID: \"dadd228d-09fb-4e5c-be66-13cdb74b60b0\") " pod="kuadrant-system/authorino-557f876f69-xwx5h"
Sep 11 18:27:08.367733 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.367708    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xwx5h"]
Sep 11 18:27:08.367906 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.367891    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xwx5h"
Sep 11 18:27:08.410171 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.410113    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d4bf65747-9bcbd"]
Sep 11 18:27:08.414760 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.414738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4bf65747-9bcbd"
Sep 11 18:27:08.421801 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.421033    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d4bf65747-9bcbd"]
Sep 11 18:27:08.529627 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.529599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7dpt\" (UniqueName: \"kubernetes.io/projected/e7d75bd6-e536-4272-ad66-0338f093b815-kube-api-access-q7dpt\") pod \"authorino-d4bf65747-9bcbd\" (UID: \"e7d75bd6-e536-4272-ad66-0338f093b815\") " pod="kuadrant-system/authorino-d4bf65747-9bcbd"
Sep 11 18:27:08.630331 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.630300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7dpt\" (UniqueName: \"kubernetes.io/projected/e7d75bd6-e536-4272-ad66-0338f093b815-kube-api-access-q7dpt\") pod \"authorino-d4bf65747-9bcbd\" (UID: \"e7d75bd6-e536-4272-ad66-0338f093b815\") " pod="kuadrant-system/authorino-d4bf65747-9bcbd"
Sep 11 18:27:08.639831 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.639795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7dpt\" (UniqueName: \"kubernetes.io/projected/e7d75bd6-e536-4272-ad66-0338f093b815-kube-api-access-q7dpt\") pod \"authorino-d4bf65747-9bcbd\" (UID: \"e7d75bd6-e536-4272-ad66-0338f093b815\") " pod="kuadrant-system/authorino-d4bf65747-9bcbd"
Sep 11 18:27:08.641266 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.641237    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xwx5h"]
Sep 11 18:27:08.644173 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:27:08.644148    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddadd228d_09fb_4e5c_be66_13cdb74b60b0.slice/crio-8d6e8688faa947e35a8c65e7060e22e171f977bdab06ac369ea3003edf034e70 WatchSource:0}: Error finding container 8d6e8688faa947e35a8c65e7060e22e171f977bdab06ac369ea3003edf034e70: Status 404 returned error can't find the container with id 8d6e8688faa947e35a8c65e7060e22e171f977bdab06ac369ea3003edf034e70
Sep 11 18:27:08.731182 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:08.731122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4bf65747-9bcbd"
Sep 11 18:27:09.011162 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.011096    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d4bf65747-9bcbd"]
Sep 11 18:27:09.014165 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:27:09.014136    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7d75bd6_e536_4272_ad66_0338f093b815.slice/crio-7140b1ea530d0c28d67b43d5494913b8426101f6f89d88037c1d9be34e0cda78 WatchSource:0}: Error finding container 7140b1ea530d0c28d67b43d5494913b8426101f6f89d88037c1d9be34e0cda78: Status 404 returned error can't find the container with id 7140b1ea530d0c28d67b43d5494913b8426101f6f89d88037c1d9be34e0cda78
Sep 11 18:27:09.610557 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.610523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-xwx5h" event={"ID":"dadd228d-09fb-4e5c-be66-13cdb74b60b0","Type":"ContainerStarted","Data":"31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e"}
Sep 11 18:27:09.610557 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.610547    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-xwx5h" podUID="dadd228d-09fb-4e5c-be66-13cdb74b60b0" containerName="authorino" containerID="cri-o://31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e" gracePeriod=30
Sep 11 18:27:09.611050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.610567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-xwx5h" event={"ID":"dadd228d-09fb-4e5c-be66-13cdb74b60b0","Type":"ContainerStarted","Data":"8d6e8688faa947e35a8c65e7060e22e171f977bdab06ac369ea3003edf034e70"}
Sep 11 18:27:09.612114 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.612092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4bf65747-9bcbd" event={"ID":"e7d75bd6-e536-4272-ad66-0338f093b815","Type":"ContainerStarted","Data":"43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce"}
Sep 11 18:27:09.612232 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.612121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4bf65747-9bcbd" event={"ID":"e7d75bd6-e536-4272-ad66-0338f093b815","Type":"ContainerStarted","Data":"7140b1ea530d0c28d67b43d5494913b8426101f6f89d88037c1d9be34e0cda78"}
Sep 11 18:27:09.627711 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.627645    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-xwx5h" podStartSLOduration=1.141069037 podStartE2EDuration="1.627631141s" totalImagesPullingTime="486.562104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:09.626514241 +0000 UTC m=+694.205778800" watchObservedRunningTime="2026-09-11 18:27:09.627631141 +0000 UTC m=+694.206895698"
Sep 11 18:27:09.645385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.645347    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d4bf65747-9bcbd" podStartSLOduration=1.230799499 podStartE2EDuration="1.645334343s" totalImagesPullingTime="414.534844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:09.644230706 +0000 UTC m=+694.223495276" watchObservedRunningTime="2026-09-11 18:27:09.645334343 +0000 UTC m=+694.224598950"
Sep 11 18:27:09.853182 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.853157    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xwx5h"
Sep 11 18:27:09.943036 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.943008    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nznfm\" (UniqueName: \"kubernetes.io/projected/dadd228d-09fb-4e5c-be66-13cdb74b60b0-kube-api-access-nznfm\") pod \"dadd228d-09fb-4e5c-be66-13cdb74b60b0\" (UID: \"dadd228d-09fb-4e5c-be66-13cdb74b60b0\") "
Sep 11 18:27:09.944887 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:09.944860    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadd228d-09fb-4e5c-be66-13cdb74b60b0-kube-api-access-nznfm" (OuterVolumeSpecName: "kube-api-access-nznfm") pod "dadd228d-09fb-4e5c-be66-13cdb74b60b0" (UID: "dadd228d-09fb-4e5c-be66-13cdb74b60b0"). InnerVolumeSpecName "kube-api-access-nznfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:27:10.043774 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:10.043752    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nznfm\" (UniqueName: \"kubernetes.io/projected/dadd228d-09fb-4e5c-be66-13cdb74b60b0-kube-api-access-nznfm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:27:10.617904 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:10.617830    2578 generic.go:356] "Generic (PLEG): container finished" podID="dadd228d-09fb-4e5c-be66-13cdb74b60b0" containerID="31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e" exitCode=0
Sep 11 18:27:10.617904 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:10.617881    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xwx5h"
Sep 11 18:27:10.618105 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:10.617915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-xwx5h" event={"ID":"dadd228d-09fb-4e5c-be66-13cdb74b60b0","Type":"ContainerDied","Data":"31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e"}
Sep 11 18:27:10.618105 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:10.617948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-xwx5h" event={"ID":"dadd228d-09fb-4e5c-be66-13cdb74b60b0","Type":"ContainerDied","Data":"8d6e8688faa947e35a8c65e7060e22e171f977bdab06ac369ea3003edf034e70"}
Sep 11 18:27:10.618105 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:10.617967    2578 scope.go:117] "RemoveContainer" containerID="31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e"
Sep 11 18:27:10.626153 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:10.626134    2578 scope.go:117] "RemoveContainer" containerID="31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e"
Sep 11 18:27:10.626404 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:10.626380    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e\": container with ID starting with 31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e not found: ID does not exist" containerID="31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e"
Sep 11 18:27:10.626445 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:10.626412    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e"} err="failed to get container status \"31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e\": rpc error: code = NotFound desc = could not find container \"31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e\": container with ID starting with 31296ee14908ba64de05235a25ac8b16ece35937bd6810ab48753a9eff02753e not found: ID does not exist"
Sep 11 18:27:10.639275 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:10.639254    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xwx5h"]
Sep 11 18:27:10.643229 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:10.643208    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-xwx5h"]
Sep 11 18:27:11.493092 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:11.493061    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4bf65747-9bcbd"]
Sep 11 18:27:11.624038 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:11.624004    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-d4bf65747-9bcbd" podUID="e7d75bd6-e536-4272-ad66-0338f093b815" containerName="authorino" containerID="cri-o://43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce" gracePeriod=30
Sep 11 18:27:11.936811 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:11.936783    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dadd228d-09fb-4e5c-be66-13cdb74b60b0" path="/var/lib/kubelet/pods/dadd228d-09fb-4e5c-be66-13cdb74b60b0/volumes"
Sep 11 18:27:12.403212 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.403190    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4bf65747-9bcbd"
Sep 11 18:27:12.566293 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.566269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7dpt\" (UniqueName: \"kubernetes.io/projected/e7d75bd6-e536-4272-ad66-0338f093b815-kube-api-access-q7dpt\") pod \"e7d75bd6-e536-4272-ad66-0338f093b815\" (UID: \"e7d75bd6-e536-4272-ad66-0338f093b815\") "
Sep 11 18:27:12.568137 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.568113    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d75bd6-e536-4272-ad66-0338f093b815-kube-api-access-q7dpt" (OuterVolumeSpecName: "kube-api-access-q7dpt") pod "e7d75bd6-e536-4272-ad66-0338f093b815" (UID: "e7d75bd6-e536-4272-ad66-0338f093b815"). InnerVolumeSpecName "kube-api-access-q7dpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:27:12.628984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.628950    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7d75bd6-e536-4272-ad66-0338f093b815" containerID="43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce" exitCode=0
Sep 11 18:27:12.629102 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.628995    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4bf65747-9bcbd"
Sep 11 18:27:12.629102 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.629034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4bf65747-9bcbd" event={"ID":"e7d75bd6-e536-4272-ad66-0338f093b815","Type":"ContainerDied","Data":"43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce"}
Sep 11 18:27:12.629102 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.629075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4bf65747-9bcbd" event={"ID":"e7d75bd6-e536-4272-ad66-0338f093b815","Type":"ContainerDied","Data":"7140b1ea530d0c28d67b43d5494913b8426101f6f89d88037c1d9be34e0cda78"}
Sep 11 18:27:12.629102 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.629093    2578 scope.go:117] "RemoveContainer" containerID="43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce"
Sep 11 18:27:12.636876 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.636858    2578 scope.go:117] "RemoveContainer" containerID="43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce"
Sep 11 18:27:12.637109 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:12.637091    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce\": container with ID starting with 43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce not found: ID does not exist" containerID="43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce"
Sep 11 18:27:12.637166 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.637116    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce"} err="failed to get container status \"43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce\": rpc error: code = NotFound desc = could not find container \"43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce\": container with ID starting with 43e384a623beb7c6d3611776f6321ef0eb4942c6ceb4b460528c8253dfc851ce not found: ID does not exist"
Sep 11 18:27:12.649800 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.649781    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4bf65747-9bcbd"]
Sep 11 18:27:12.653556 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.653539    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d4bf65747-9bcbd"]
Sep 11 18:27:12.667601 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:12.667584    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7dpt\" (UniqueName: \"kubernetes.io/projected/e7d75bd6-e536-4272-ad66-0338f093b815-kube-api-access-q7dpt\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:27:13.936890 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:13.936824    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d75bd6-e536-4272-ad66-0338f093b815" path="/var/lib/kubelet/pods/e7d75bd6-e536-4272-ad66-0338f093b815/volumes"
Sep 11 18:27:14.661194 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.661156    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7cccd8676f-hdmz2"]
Sep 11 18:27:14.661526 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.661511    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadd228d-09fb-4e5c-be66-13cdb74b60b0" containerName="authorino"
Sep 11 18:27:14.661609 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.661530    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadd228d-09fb-4e5c-be66-13cdb74b60b0" containerName="authorino"
Sep 11 18:27:14.661609 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.661549    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d75bd6-e536-4272-ad66-0338f093b815" containerName="authorino"
Sep 11 18:27:14.661609 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.661557    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d75bd6-e536-4272-ad66-0338f093b815" containerName="authorino"
Sep 11 18:27:14.661788 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.661643    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dadd228d-09fb-4e5c-be66-13cdb74b60b0" containerName="authorino"
Sep 11 18:27:14.661788 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.661678    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d75bd6-e536-4272-ad66-0338f093b815" containerName="authorino"
Sep 11 18:27:14.664524 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.664500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:14.667197 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.667176    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:27:14.667326 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.667205    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 18:27:14.667326 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.667269    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7x2vx\""
Sep 11 18:27:14.670001 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.669984    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 18:27:14.678503 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.678482    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cccd8676f-hdmz2"]
Sep 11 18:27:14.782786 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.782764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-webhook-cert\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:14.782880 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.782809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-metrics-certs\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:14.782921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.782877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xcn8\" (UniqueName: \"kubernetes.io/projected/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-kube-api-access-7xcn8\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:14.824021 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.823997    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84c45b5c4-9lrq9"]
Sep 11 18:27:14.826202 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.826188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:14.841676 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.841636    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-9lrq9"]
Sep 11 18:27:14.883274 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.883242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xcn8\" (UniqueName: \"kubernetes.io/projected/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-kube-api-access-7xcn8\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:14.883376 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.883307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-webhook-cert\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:14.883376 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.883354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-metrics-certs\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:14.883486 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:14.883465    2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 18:27:14.883486 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:14.883467    2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 11 18:27:14.883563 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:14.883523    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-metrics-certs podName:b1fa56c7-ecfa-42c8-b08d-212e84ec8080 nodeName:}" failed. No retries permitted until 2026-09-11 18:27:15.383504054 +0000 UTC m=+699.962768599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-metrics-certs") pod "maas-controller-7cccd8676f-hdmz2" (UID: "b1fa56c7-ecfa-42c8-b08d-212e84ec8080") : secret "maas-controller-metrics-tls" not found
Sep 11 18:27:14.883563 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:14.883539    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-webhook-cert podName:b1fa56c7-ecfa-42c8-b08d-212e84ec8080 nodeName:}" failed. No retries permitted until 2026-09-11 18:27:15.383530333 +0000 UTC m=+699.962794875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-webhook-cert") pod "maas-controller-7cccd8676f-hdmz2" (UID: "b1fa56c7-ecfa-42c8-b08d-212e84ec8080") : secret "maas-controller-webhook-cert" not found
Sep 11 18:27:14.894428 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.894404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xcn8\" (UniqueName: \"kubernetes.io/projected/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-kube-api-access-7xcn8\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:14.984455 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.984399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-webhook-cert\") pod \"maas-controller-84c45b5c4-9lrq9\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") " pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:14.984455 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.984434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v86tb\" (UniqueName: \"kubernetes.io/projected/f0321f86-75ec-4948-b3e1-81dafebb5c20-kube-api-access-v86tb\") pod \"maas-controller-84c45b5c4-9lrq9\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") " pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:14.984803 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:14.984459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-metrics-certs\") pod \"maas-controller-84c45b5c4-9lrq9\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") " pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:15.085061 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.085035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-webhook-cert\") pod \"maas-controller-84c45b5c4-9lrq9\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") " pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:15.085150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.085077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v86tb\" (UniqueName: \"kubernetes.io/projected/f0321f86-75ec-4948-b3e1-81dafebb5c20-kube-api-access-v86tb\") pod \"maas-controller-84c45b5c4-9lrq9\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") " pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:15.085150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.085102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-metrics-certs\") pod \"maas-controller-84c45b5c4-9lrq9\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") " pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:15.085217 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:15.085207    2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 18:27:15.085268 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:15.085257    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-metrics-certs podName:f0321f86-75ec-4948-b3e1-81dafebb5c20 nodeName:}" failed. No retries permitted until 2026-09-11 18:27:15.585241281 +0000 UTC m=+700.164505833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-metrics-certs") pod "maas-controller-84c45b5c4-9lrq9" (UID: "f0321f86-75ec-4948-b3e1-81dafebb5c20") : secret "maas-controller-metrics-tls" not found
Sep 11 18:27:15.087469 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.087443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-webhook-cert\") pod \"maas-controller-84c45b5c4-9lrq9\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") " pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:15.094342 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.094324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v86tb\" (UniqueName: \"kubernetes.io/projected/f0321f86-75ec-4948-b3e1-81dafebb5c20-kube-api-access-v86tb\") pod \"maas-controller-84c45b5c4-9lrq9\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") " pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:15.386351 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.386316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-webhook-cert\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:15.386503 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.386377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-metrics-certs\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:15.388724 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.388701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-metrics-certs\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:15.388837 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.388750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-webhook-cert\") pod \"maas-controller-7cccd8676f-hdmz2\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") " pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:15.576374 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.576349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:15.587638 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.587607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-metrics-certs\") pod \"maas-controller-84c45b5c4-9lrq9\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") " pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:15.589928 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.589904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-metrics-certs\") pod \"maas-controller-84c45b5c4-9lrq9\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") " pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:15.736894 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.736826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:15.844913 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:15.844893    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cccd8676f-hdmz2"]
Sep 11 18:27:15.847178 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:27:15.847154    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fa56c7_ecfa_42c8_b08d_212e84ec8080.slice/crio-64db296238fe627ae59e76abef746a23be049e666b35e1b77ab5287abcd4fb8c WatchSource:0}: Error finding container 64db296238fe627ae59e76abef746a23be049e666b35e1b77ab5287abcd4fb8c: Status 404 returned error can't find the container with id 64db296238fe627ae59e76abef746a23be049e666b35e1b77ab5287abcd4fb8c
Sep 11 18:27:16.005611 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:16.005590    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-9lrq9"]
Sep 11 18:27:16.008238 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:27:16.008213    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0321f86_75ec_4948_b3e1_81dafebb5c20.slice/crio-987ffb7473c265c9877e6ee50515609acf65f841e6b2b137ab761e4a3293da58 WatchSource:0}: Error finding container 987ffb7473c265c9877e6ee50515609acf65f841e6b2b137ab761e4a3293da58: Status 404 returned error can't find the container with id 987ffb7473c265c9877e6ee50515609acf65f841e6b2b137ab761e4a3293da58
Sep 11 18:27:16.653817 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:16.653714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c45b5c4-9lrq9" event={"ID":"f0321f86-75ec-4948-b3e1-81dafebb5c20","Type":"ContainerStarted","Data":"987ffb7473c265c9877e6ee50515609acf65f841e6b2b137ab761e4a3293da58"}
Sep 11 18:27:16.656287 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:16.656209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cccd8676f-hdmz2" event={"ID":"b1fa56c7-ecfa-42c8-b08d-212e84ec8080","Type":"ContainerStarted","Data":"64db296238fe627ae59e76abef746a23be049e666b35e1b77ab5287abcd4fb8c"}
Sep 11 18:27:18.669210 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:18.669172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cccd8676f-hdmz2" event={"ID":"b1fa56c7-ecfa-42c8-b08d-212e84ec8080","Type":"ContainerStarted","Data":"81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac"}
Sep 11 18:27:18.669582 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:18.669239    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:18.670954 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:18.670931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c45b5c4-9lrq9" event={"ID":"f0321f86-75ec-4948-b3e1-81dafebb5c20","Type":"ContainerStarted","Data":"f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2"}
Sep 11 18:27:18.671072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:18.671059    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:18.696158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:18.696091    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7cccd8676f-hdmz2" podStartSLOduration=2.026109399 podStartE2EDuration="4.696076479s" totalImagesPullingTime="2.66996708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:18.695181574 +0000 UTC m=+703.274446134" watchObservedRunningTime="2026-09-11 18:27:18.696076479 +0000 UTC m=+703.275341038"
Sep 11 18:27:18.739224 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:18.739170    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84c45b5c4-9lrq9" podStartSLOduration=2.220622483 podStartE2EDuration="4.739153079s" totalImagesPullingTime="2.518530596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:18.737007146 +0000 UTC m=+703.316271706" watchObservedRunningTime="2026-09-11 18:27:18.739153079 +0000 UTC m=+703.318417638"
Sep 11 18:27:29.677613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:29.677541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:27:29.677613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:29.677610    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:29.762329 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:29.762296    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cccd8676f-hdmz2"]
Sep 11 18:27:29.762542 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:29.762515    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7cccd8676f-hdmz2" podUID="b1fa56c7-ecfa-42c8-b08d-212e84ec8080" containerName="manager" containerID="cri-o://81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac" gracePeriod=30
Sep 11 18:27:30.002241 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.002219    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:30.099960 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.099928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-metrics-certs\") pod \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") "
Sep 11 18:27:30.100101 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.099987    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-webhook-cert\") pod \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") "
Sep 11 18:27:30.100101 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.100017    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xcn8\" (UniqueName: \"kubernetes.io/projected/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-kube-api-access-7xcn8\") pod \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\" (UID: \"b1fa56c7-ecfa-42c8-b08d-212e84ec8080\") "
Sep 11 18:27:30.102074 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.102033    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b1fa56c7-ecfa-42c8-b08d-212e84ec8080" (UID: "b1fa56c7-ecfa-42c8-b08d-212e84ec8080"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:27:30.102172 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.102086    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b1fa56c7-ecfa-42c8-b08d-212e84ec8080" (UID: "b1fa56c7-ecfa-42c8-b08d-212e84ec8080"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:27:30.102172 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.102156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-kube-api-access-7xcn8" (OuterVolumeSpecName: "kube-api-access-7xcn8") pod "b1fa56c7-ecfa-42c8-b08d-212e84ec8080" (UID: "b1fa56c7-ecfa-42c8-b08d-212e84ec8080"). InnerVolumeSpecName "kube-api-access-7xcn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:27:30.200666 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.200628    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-webhook-cert\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:27:30.200756 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.200667    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xcn8\" (UniqueName: \"kubernetes.io/projected/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-kube-api-access-7xcn8\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:27:30.200756 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.200680    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1fa56c7-ecfa-42c8-b08d-212e84ec8080-metrics-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:27:30.720507 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.720478    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1fa56c7-ecfa-42c8-b08d-212e84ec8080" containerID="81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac" exitCode=0
Sep 11 18:27:30.720890 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.720516    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cccd8676f-hdmz2"
Sep 11 18:27:30.720890 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.720541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cccd8676f-hdmz2" event={"ID":"b1fa56c7-ecfa-42c8-b08d-212e84ec8080","Type":"ContainerDied","Data":"81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac"}
Sep 11 18:27:30.720890 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.720585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cccd8676f-hdmz2" event={"ID":"b1fa56c7-ecfa-42c8-b08d-212e84ec8080","Type":"ContainerDied","Data":"64db296238fe627ae59e76abef746a23be049e666b35e1b77ab5287abcd4fb8c"}
Sep 11 18:27:30.720890 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.720608    2578 scope.go:117] "RemoveContainer" containerID="81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac"
Sep 11 18:27:30.729050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.729035    2578 scope.go:117] "RemoveContainer" containerID="81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac"
Sep 11 18:27:30.729322 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:30.729304    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac\": container with ID starting with 81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac not found: ID does not exist" containerID="81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac"
Sep 11 18:27:30.729366 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.729330    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac"} err="failed to get container status \"81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac\": rpc error: code = NotFound desc = could not find container \"81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac\": container with ID starting with 81a34269314a0d7ae69f7dffd9c1c1a3b8eb3530b630ab7be40f3769970cb8ac not found: ID does not exist"
Sep 11 18:27:30.742626 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.742598    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cccd8676f-hdmz2"]
Sep 11 18:27:30.746662 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:30.746624    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7cccd8676f-hdmz2"]
Sep 11 18:27:31.936872 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:31.936822    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1fa56c7-ecfa-42c8-b08d-212e84ec8080" path="/var/lib/kubelet/pods/b1fa56c7-ecfa-42c8-b08d-212e84ec8080/volumes"
Sep 11 18:27:51.973306 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:51.973269    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-b97cd"]
Sep 11 18:27:51.974555 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:51.974527    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fa56c7-ecfa-42c8-b08d-212e84ec8080" containerName="manager"
Sep 11 18:27:51.974784 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:51.974770    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fa56c7-ecfa-42c8-b08d-212e84ec8080" containerName="manager"
Sep 11 18:27:51.975011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:51.974999    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fa56c7-ecfa-42c8-b08d-212e84ec8080" containerName="manager"
Sep 11 18:27:51.980272 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:51.980249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:51.987058 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:51.987038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l7zn9\""
Sep 11 18:27:51.987156 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:51.987058    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 18:27:51.987156 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:51.987045    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:27:51.993600 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:51.993580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-b97cd"]
Sep 11 18:27:52.055021 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.054996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-maas-api-tls\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.055142 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.055024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm6sf\" (UniqueName: \"kubernetes.io/projected/161921c5-e475-41b2-a9e9-dc711195e82a-kube-api-access-cm6sf\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.055142 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.055042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/161921c5-e475-41b2-a9e9-dc711195e82a-openshift-service-ca\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.055224 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.055151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-metrics-certs\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.156251 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.156221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-maas-api-tls\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.156251 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.156254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm6sf\" (UniqueName: \"kubernetes.io/projected/161921c5-e475-41b2-a9e9-dc711195e82a-kube-api-access-cm6sf\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.156392 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.156270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/161921c5-e475-41b2-a9e9-dc711195e82a-openshift-service-ca\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.156392 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.156340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-metrics-certs\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.156455 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:52.156447    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 11 18:27:52.156555 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:27:52.156545    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-metrics-certs podName:161921c5-e475-41b2-a9e9-dc711195e82a nodeName:}" failed. No retries permitted until 2026-09-11 18:27:52.656523737 +0000 UTC m=+737.235788278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-metrics-certs") pod "maas-api-976f486c4-b97cd" (UID: "161921c5-e475-41b2-a9e9-dc711195e82a") : secret "maas-api-metrics-tls" not found
Sep 11 18:27:52.157110 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.157084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/161921c5-e475-41b2-a9e9-dc711195e82a-openshift-service-ca\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.158585 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.158562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-maas-api-tls\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.166363 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.166340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm6sf\" (UniqueName: \"kubernetes.io/projected/161921c5-e475-41b2-a9e9-dc711195e82a-kube-api-access-cm6sf\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.613762 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.613730    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"]
Sep 11 18:27:52.616054 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.616034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"
Sep 11 18:27:52.619420 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.619403    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 18:27:52.619629 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.619614    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-knj28\""
Sep 11 18:27:52.631018 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.630998    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"]
Sep 11 18:27:52.661165 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.661140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4n7\" (UniqueName: \"kubernetes.io/projected/2a4ae1ca-0b57-492c-b6f1-dba25973b579-kube-api-access-9b4n7\") pod \"payload-pre-processing-59fbc47bf8-mh2b4\" (UID: \"2a4ae1ca-0b57-492c-b6f1-dba25973b579\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"
Sep 11 18:27:52.661265 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.661179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-metrics-certs\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.661265 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.661246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4ae1ca-0b57-492c-b6f1-dba25973b579-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-mh2b4\" (UID: \"2a4ae1ca-0b57-492c-b6f1-dba25973b579\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"
Sep 11 18:27:52.663263 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.663246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-metrics-certs\") pod \"maas-api-976f486c4-b97cd\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") " pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.762408 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.762374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4n7\" (UniqueName: \"kubernetes.io/projected/2a4ae1ca-0b57-492c-b6f1-dba25973b579-kube-api-access-9b4n7\") pod \"payload-pre-processing-59fbc47bf8-mh2b4\" (UID: \"2a4ae1ca-0b57-492c-b6f1-dba25973b579\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"
Sep 11 18:27:52.762522 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.762432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4ae1ca-0b57-492c-b6f1-dba25973b579-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-mh2b4\" (UID: \"2a4ae1ca-0b57-492c-b6f1-dba25973b579\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"
Sep 11 18:27:52.763030 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.763013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4ae1ca-0b57-492c-b6f1-dba25973b579-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-mh2b4\" (UID: \"2a4ae1ca-0b57-492c-b6f1-dba25973b579\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"
Sep 11 18:27:52.775031 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.775005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4n7\" (UniqueName: \"kubernetes.io/projected/2a4ae1ca-0b57-492c-b6f1-dba25973b579-kube-api-access-9b4n7\") pod \"payload-pre-processing-59fbc47bf8-mh2b4\" (UID: \"2a4ae1ca-0b57-492c-b6f1-dba25973b579\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"
Sep 11 18:27:52.890097 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.890020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:52.925614 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:52.925576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"
Sep 11 18:27:53.175631 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:53.175600    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-b97cd"]
Sep 11 18:27:53.177919 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:27:53.177888    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161921c5_e475_41b2_a9e9_dc711195e82a.slice/crio-3ef52d4d7f6fca79693f9fbb49ed7c3d6fc68d1c0e664b97c46a75a06a944e99 WatchSource:0}: Error finding container 3ef52d4d7f6fca79693f9fbb49ed7c3d6fc68d1c0e664b97c46a75a06a944e99: Status 404 returned error can't find the container with id 3ef52d4d7f6fca79693f9fbb49ed7c3d6fc68d1c0e664b97c46a75a06a944e99
Sep 11 18:27:53.204604 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:53.204583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"]
Sep 11 18:27:53.205828 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:27:53.205803    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4ae1ca_0b57_492c_b6f1_dba25973b579.slice/crio-16174081f81a6570c2951b2ef757262e9206b1d124b620845c5b75c9e736c70b WatchSource:0}: Error finding container 16174081f81a6570c2951b2ef757262e9206b1d124b620845c5b75c9e736c70b: Status 404 returned error can't find the container with id 16174081f81a6570c2951b2ef757262e9206b1d124b620845c5b75c9e736c70b
Sep 11 18:27:53.826030 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:53.825963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd" event={"ID":"161921c5-e475-41b2-a9e9-dc711195e82a","Type":"ContainerStarted","Data":"3ef52d4d7f6fca79693f9fbb49ed7c3d6fc68d1c0e664b97c46a75a06a944e99"}
Sep 11 18:27:53.828037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:53.828008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4" event={"ID":"2a4ae1ca-0b57-492c-b6f1-dba25973b579","Type":"ContainerStarted","Data":"16174081f81a6570c2951b2ef757262e9206b1d124b620845c5b75c9e736c70b"}
Sep 11 18:27:56.847402 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:56.847361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd" event={"ID":"161921c5-e475-41b2-a9e9-dc711195e82a","Type":"ContainerStarted","Data":"98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c"}
Sep 11 18:27:56.847849 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:56.847473    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:27:56.848948 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:56.848925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4" event={"ID":"2a4ae1ca-0b57-492c-b6f1-dba25973b579","Type":"ContainerStarted","Data":"b0f98a5fe3b5f9b072660f5c197ecaf50cf16365bc7adf7d1aa0c1eb0484ffc1"}
Sep 11 18:27:56.849065 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:56.849023    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"
Sep 11 18:27:56.870154 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:56.870117    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd" podStartSLOduration=3.258181211 podStartE2EDuration="5.870103629s" totalImagesPullingTime="2.611922418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:56.867907726 +0000 UTC m=+741.447172289" watchObservedRunningTime="2026-09-11 18:27:56.870103629 +0000 UTC m=+741.449368187"
Sep 11 18:27:56.887434 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:27:56.887391    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4" podStartSLOduration=2.274884516 podStartE2EDuration="4.887380678s" totalImagesPullingTime="2.612496162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:56.886046509 +0000 UTC m=+741.465311080" watchObservedRunningTime="2026-09-11 18:27:56.887380678 +0000 UTC m=+741.466645237"
Sep 11 18:28:02.858692 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:02.858665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 18:28:07.855370 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:07.855341    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mh2b4"
Sep 11 18:28:27.251448 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.251416    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"]
Sep 11 18:28:27.256308 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.256287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.260811 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.260790    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x8pc2\""
Sep 11 18:28:27.260934 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.260813    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 18:28:27.260934 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.260828    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 18:28:27.260934 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.260837    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 18:28:27.267437 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.267418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"]
Sep 11 18:28:27.418817 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.418789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.418969 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.418823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.418969 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.418850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.418969 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.418871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npwwt\" (UniqueName: \"kubernetes.io/projected/819416f3-065f-430e-b277-8f718459feac-kube-api-access-npwwt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.418969 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.418904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.418969 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.418947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819416f3-065f-430e-b277-8f718459feac-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.519747 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.519680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819416f3-065f-430e-b277-8f718459feac-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.519853 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.519773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.519853 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.519801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.519853 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.519827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.519997 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.519850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npwwt\" (UniqueName: \"kubernetes.io/projected/819416f3-065f-430e-b277-8f718459feac-kube-api-access-npwwt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.519997 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.519884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.520424 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.520402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.520527 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.520431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.520527 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.520499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.522033 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.522015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819416f3-065f-430e-b277-8f718459feac-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.522147 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.522131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819416f3-065f-430e-b277-8f718459feac-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.529548 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.529528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npwwt\" (UniqueName: \"kubernetes.io/projected/819416f3-065f-430e-b277-8f718459feac-kube-api-access-npwwt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sjkxb\" (UID: \"819416f3-065f-430e-b277-8f718459feac\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.566504 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.566483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:27.843412 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.843390    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"]
Sep 11 18:28:27.845174 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:28:27.845134    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819416f3_065f_430e_b277_8f718459feac.slice/crio-e3e974878627489d96416e075d3862c68fbb2a37638d7207d4c4c2bfa713c1b0 WatchSource:0}: Error finding container e3e974878627489d96416e075d3862c68fbb2a37638d7207d4c4c2bfa713c1b0: Status 404 returned error can't find the container with id e3e974878627489d96416e075d3862c68fbb2a37638d7207d4c4c2bfa713c1b0
Sep 11 18:28:27.846847 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.846831    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:28:27.981077 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:27.981043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb" event={"ID":"819416f3-065f-430e-b277-8f718459feac","Type":"ContainerStarted","Data":"e3e974878627489d96416e075d3862c68fbb2a37638d7207d4c4c2bfa713c1b0"}
Sep 11 18:28:28.987676 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:28.987627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb" event={"ID":"819416f3-065f-430e-b277-8f718459feac","Type":"ContainerStarted","Data":"974084bbd9b507698c5249b9b854a4b2203b2d224a35f10d75dca786a17fbd15"}
Sep 11 18:28:29.014374 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:29.014329    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb" podStartSLOduration=1.034326714 podStartE2EDuration="2.014314225s" totalImagesPullingTime="979.987511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:29.0122301 +0000 UTC m=+773.591494752" watchObservedRunningTime="2026-09-11 18:28:29.014314225 +0000 UTC m=+773.593578786"
Sep 11 18:28:37.567288 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:37.567242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:37.568072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:37.567322    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:37.579560 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:37.579542    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:38.034612 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:38.034586    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sjkxb"
Sep 11 18:28:47.557073 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.557033    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"]
Sep 11 18:28:47.559703 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.559668    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.563504 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.563482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 18:28:47.567100 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.567025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.567100 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.567061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.567100 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.567091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.567260 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.567114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hpvr\" (UniqueName: \"kubernetes.io/projected/54b6cb06-0c4b-4825-9dac-9b32336a882c-kube-api-access-8hpvr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.567260 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.567158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.567260 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.567241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54b6cb06-0c4b-4825-9dac-9b32336a882c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.579445 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.579422    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"]
Sep 11 18:28:47.667860 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.667834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54b6cb06-0c4b-4825-9dac-9b32336a882c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.668008 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.667876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.668008 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.667893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.668008 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.667910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.668144 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.668013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hpvr\" (UniqueName: \"kubernetes.io/projected/54b6cb06-0c4b-4825-9dac-9b32336a882c-kube-api-access-8hpvr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.668144 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.668049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.668293 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.668266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.668353 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.668278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.668405 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.668385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.670018 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.669992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54b6cb06-0c4b-4825-9dac-9b32336a882c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.670195 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.670177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54b6cb06-0c4b-4825-9dac-9b32336a882c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.677440 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.677419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hpvr\" (UniqueName: \"kubernetes.io/projected/54b6cb06-0c4b-4825-9dac-9b32336a882c-kube-api-access-8hpvr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm\" (UID: \"54b6cb06-0c4b-4825-9dac-9b32336a882c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:47.870935 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:47.870875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:48.145241 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:48.145215    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"]
Sep 11 18:28:48.147434 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:28:48.147405    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b6cb06_0c4b_4825_9dac_9b32336a882c.slice/crio-76db7132db74e2dfe00401bdb9a10922e4f4756895418fc3519b384856078071 WatchSource:0}: Error finding container 76db7132db74e2dfe00401bdb9a10922e4f4756895418fc3519b384856078071: Status 404 returned error can't find the container with id 76db7132db74e2dfe00401bdb9a10922e4f4756895418fc3519b384856078071
Sep 11 18:28:49.069176 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:49.069141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm" event={"ID":"54b6cb06-0c4b-4825-9dac-9b32336a882c","Type":"ContainerStarted","Data":"c64c131da5966c7c359a0b7ee79ddb0ad02a8530982ed32fd222b7fc8f9a9040"}
Sep 11 18:28:49.069525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:49.069192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm" event={"ID":"54b6cb06-0c4b-4825-9dac-9b32336a882c","Type":"ContainerStarted","Data":"76db7132db74e2dfe00401bdb9a10922e4f4756895418fc3519b384856078071"}
Sep 11 18:28:49.095329 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:49.095283    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm" podStartSLOduration=1.9097990820000001 podStartE2EDuration="2.095268269s" totalImagesPullingTime="185.469187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:49.09168362 +0000 UTC m=+793.670948181" watchObservedRunningTime="2026-09-11 18:28:49.095268269 +0000 UTC m=+793.674532828"
Sep 11 18:28:57.871403 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:57.871373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:57.871776 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:57.871457    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:57.883262 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:57.883239    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:28:58.118819 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:28:58.118752    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm"
Sep 11 18:29:08.036244 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.036211    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"]
Sep 11 18:29:08.039880 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.039858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.043187 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.043169    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 18:29:08.054032 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.054012    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"]
Sep 11 18:29:08.119366 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.119344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.119467 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.119371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66397def-0642-484c-9ad9-b0c635ee8385-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.119467 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.119447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgz86\" (UniqueName: \"kubernetes.io/projected/66397def-0642-484c-9ad9-b0c635ee8385-kube-api-access-pgz86\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.119536 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.119472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.119536 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.119506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.119601 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.119535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.220028 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.219999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgz86\" (UniqueName: \"kubernetes.io/projected/66397def-0642-484c-9ad9-b0c635ee8385-kube-api-access-pgz86\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.220028 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.220028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.220208 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.220056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.220208 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.220078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.220208 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.220113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.220208 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.220133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66397def-0642-484c-9ad9-b0c635ee8385-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.220428 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.220406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.220493 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.220457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.220591 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.220573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.222218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.222201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66397def-0642-484c-9ad9-b0c635ee8385-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.222307 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.222289    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66397def-0642-484c-9ad9-b0c635ee8385-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.228016 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.227999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgz86\" (UniqueName: \"kubernetes.io/projected/66397def-0642-484c-9ad9-b0c635ee8385-kube-api-access-pgz86\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj\" (UID: \"66397def-0642-484c-9ad9-b0c635ee8385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.349715 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.349643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:08.620385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:08.620313    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"]
Sep 11 18:29:08.623102 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:29:08.623076    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66397def_0642_484c_9ad9_b0c635ee8385.slice/crio-9d582ecbadaa43a0be4901679e7b0ba6dbeaca0e3fb8e1bfdcb77fdb5bc601a8 WatchSource:0}: Error finding container 9d582ecbadaa43a0be4901679e7b0ba6dbeaca0e3fb8e1bfdcb77fdb5bc601a8: Status 404 returned error can't find the container with id 9d582ecbadaa43a0be4901679e7b0ba6dbeaca0e3fb8e1bfdcb77fdb5bc601a8
Sep 11 18:29:09.164837 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:09.164798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj" event={"ID":"66397def-0642-484c-9ad9-b0c635ee8385","Type":"ContainerStarted","Data":"39feab4a212f7382778c909fa4e3ce2cab0d3103363da1b9a5ff6f3975a0787f"}
Sep 11 18:29:09.165209 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:09.164853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj" event={"ID":"66397def-0642-484c-9ad9-b0c635ee8385","Type":"ContainerStarted","Data":"9d582ecbadaa43a0be4901679e7b0ba6dbeaca0e3fb8e1bfdcb77fdb5bc601a8"}
Sep 11 18:29:09.186121 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:09.186074    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj" podStartSLOduration=0.998893877 podStartE2EDuration="1.186060019s" totalImagesPullingTime="187.166142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:09.184282565 +0000 UTC m=+813.763547126" watchObservedRunningTime="2026-09-11 18:29:09.186060019 +0000 UTC m=+813.765324578"
Sep 11 18:29:18.350386 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:18.350356    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:18.350830 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:18.350440    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:18.362348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:18.362323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:29:19.216832 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:29:19.216797    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj"
Sep 11 18:32:02.684720 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.684627    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"]
Sep 11 18:32:02.687397 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.687374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.689872 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.689851    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4d5a75-serving-cert\""
Sep 11 18:32:02.699972 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.699948    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"]
Sep 11 18:32:02.737425 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.737392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qwl9\" (UniqueName: \"kubernetes.io/projected/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-kube-api-access-8qwl9\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.737596 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.737438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-metrics-certs\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.737596 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.737501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.737596 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.737549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.838623 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.838587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.838805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.838671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwl9\" (UniqueName: \"kubernetes.io/projected/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-kube-api-access-8qwl9\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.838805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.838713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-metrics-certs\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.838805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.838773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.839381 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.839353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.841217 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.841194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-metrics-certs\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.841306 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.841200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.847196 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.847170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwl9\" (UniqueName: \"kubernetes.io/projected/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-kube-api-access-8qwl9\") pod \"maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:02.998484 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:02.998400    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:03.270812 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.270783    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"]
Sep 11 18:32:03.272892 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:32:03.272860    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d770dc7_4c9a_4e0a_aba5_92a88f2f8ceb.slice/crio-b71db1aa02b473e0ae724f95683eff4271d78b9b60ea6f328ae2315e78aee311 WatchSource:0}: Error finding container b71db1aa02b473e0ae724f95683eff4271d78b9b60ea6f328ae2315e78aee311: Status 404 returned error can't find the container with id b71db1aa02b473e0ae724f95683eff4271d78b9b60ea6f328ae2315e78aee311
Sep 11 18:32:03.516666 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.516620    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"]
Sep 11 18:32:03.519225 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.519202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:32:03.522128 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.522093    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4d5a75\""
Sep 11 18:32:03.522242 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.522177    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4d5a75-dockercfg-679mc\""
Sep 11 18:32:03.533924 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.533899    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"]
Sep 11 18:32:03.545435 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.545279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0deda4de-c01f-4161-bdbe-33dcafa6e04d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq\" (UID: \"0deda4de-c01f-4161-bdbe-33dcafa6e04d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:32:03.545435 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.545329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h44g\" (UniqueName: \"kubernetes.io/projected/0deda4de-c01f-4161-bdbe-33dcafa6e04d-kube-api-access-9h44g\") pod \"payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq\" (UID: \"0deda4de-c01f-4161-bdbe-33dcafa6e04d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:32:03.559764 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.559740    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"]
Sep 11 18:32:03.562568 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.562550    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:32:03.575170 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.575147    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"]
Sep 11 18:32:03.646446 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.646415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0deda4de-c01f-4161-bdbe-33dcafa6e04d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq\" (UID: \"0deda4de-c01f-4161-bdbe-33dcafa6e04d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:32:03.646627 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.646453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h44g\" (UniqueName: \"kubernetes.io/projected/0deda4de-c01f-4161-bdbe-33dcafa6e04d-kube-api-access-9h44g\") pod \"payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq\" (UID: \"0deda4de-c01f-4161-bdbe-33dcafa6e04d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:32:03.646627 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.646553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjn5\" (UniqueName: \"kubernetes.io/projected/b3980cc4-62a8-4e38-a426-870184f86296-kube-api-access-9gjn5\") pod \"payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp\" (UID: \"b3980cc4-62a8-4e38-a426-870184f86296\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:32:03.646627 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.646597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3980cc4-62a8-4e38-a426-870184f86296-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp\" (UID: \"b3980cc4-62a8-4e38-a426-870184f86296\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:32:03.647104 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.647087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0deda4de-c01f-4161-bdbe-33dcafa6e04d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq\" (UID: \"0deda4de-c01f-4161-bdbe-33dcafa6e04d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:32:03.659382 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.659360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h44g\" (UniqueName: \"kubernetes.io/projected/0deda4de-c01f-4161-bdbe-33dcafa6e04d-kube-api-access-9h44g\") pod \"payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq\" (UID: \"0deda4de-c01f-4161-bdbe-33dcafa6e04d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:32:03.747881 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.747851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjn5\" (UniqueName: \"kubernetes.io/projected/b3980cc4-62a8-4e38-a426-870184f86296-kube-api-access-9gjn5\") pod \"payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp\" (UID: \"b3980cc4-62a8-4e38-a426-870184f86296\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:32:03.748165 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.747914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3980cc4-62a8-4e38-a426-870184f86296-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp\" (UID: \"b3980cc4-62a8-4e38-a426-870184f86296\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:32:03.748758 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.748736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3980cc4-62a8-4e38-a426-870184f86296-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp\" (UID: \"b3980cc4-62a8-4e38-a426-870184f86296\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:32:03.755531 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.755512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjn5\" (UniqueName: \"kubernetes.io/projected/b3980cc4-62a8-4e38-a426-870184f86296-kube-api-access-9gjn5\") pod \"payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp\" (UID: \"b3980cc4-62a8-4e38-a426-870184f86296\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:32:03.830572 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.830541    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:32:03.872568 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.872539    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:32:03.950522 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.950461    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:03.950709 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.950516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c" event={"ID":"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb","Type":"ContainerStarted","Data":"b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16"}
Sep 11 18:32:03.950709 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.950561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c" event={"ID":"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb","Type":"ContainerStarted","Data":"b71db1aa02b473e0ae724f95683eff4271d78b9b60ea6f328ae2315e78aee311"}
Sep 11 18:32:03.964943 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:03.964690    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c" podStartSLOduration=1.506352509 podStartE2EDuration="1.964642894s" totalImagesPullingTime="458.290385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:03.963671172 +0000 UTC m=+988.542935724" watchObservedRunningTime="2026-09-11 18:32:03.964642894 +0000 UTC m=+988.543907455"
Sep 11 18:32:04.144165 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:32:04.144132    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0deda4de_c01f_4161_bdbe_33dcafa6e04d.slice/crio-e2389e455f10faa23d12a78a8f788549513565fc0d4652ddd5f47bbbd9595c2c WatchSource:0}: Error finding container e2389e455f10faa23d12a78a8f788549513565fc0d4652ddd5f47bbbd9595c2c: Status 404 returned error can't find the container with id e2389e455f10faa23d12a78a8f788549513565fc0d4652ddd5f47bbbd9595c2c
Sep 11 18:32:04.146579 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.146555    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"]
Sep 11 18:32:04.176050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.176028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"]
Sep 11 18:32:04.181745 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:32:04.181717    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3980cc4_62a8_4e38_a426_870184f86296.slice/crio-62dfa91ff03b974daeb0861fa05bcadff6dd86302c33a51c893cef67b84df1ff WatchSource:0}: Error finding container 62dfa91ff03b974daeb0861fa05bcadff6dd86302c33a51c893cef67b84df1ff: Status 404 returned error can't find the container with id 62dfa91ff03b974daeb0861fa05bcadff6dd86302c33a51c893cef67b84df1ff
Sep 11 18:32:04.677662 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.677620    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"]
Sep 11 18:32:04.680363 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.680347    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:32:04.683169 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.683147    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-200c07-dockercfg-qvcmm\""
Sep 11 18:32:04.683293 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.683149    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-200c07\""
Sep 11 18:32:04.693880 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.693856    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"]
Sep 11 18:32:04.758402 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.758363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjdvd\" (UniqueName: \"kubernetes.io/projected/2b2fa513-81f4-46f1-aec8-e1aa167ef874-kube-api-access-jjdvd\") pod \"payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq\" (UID: \"2b2fa513-81f4-46f1-aec8-e1aa167ef874\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:32:04.758833 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.758433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2fa513-81f4-46f1-aec8-e1aa167ef874-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq\" (UID: \"2b2fa513-81f4-46f1-aec8-e1aa167ef874\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:32:04.860069 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.860025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdvd\" (UniqueName: \"kubernetes.io/projected/2b2fa513-81f4-46f1-aec8-e1aa167ef874-kube-api-access-jjdvd\") pod \"payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq\" (UID: \"2b2fa513-81f4-46f1-aec8-e1aa167ef874\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:32:04.860235 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.860087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2fa513-81f4-46f1-aec8-e1aa167ef874-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq\" (UID: \"2b2fa513-81f4-46f1-aec8-e1aa167ef874\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:32:04.860840 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.860818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2fa513-81f4-46f1-aec8-e1aa167ef874-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq\" (UID: \"2b2fa513-81f4-46f1-aec8-e1aa167ef874\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:32:04.872437 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.872413    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"]
Sep 11 18:32:04.875036 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.874995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdvd\" (UniqueName: \"kubernetes.io/projected/2b2fa513-81f4-46f1-aec8-e1aa167ef874-kube-api-access-jjdvd\") pod \"payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq\" (UID: \"2b2fa513-81f4-46f1-aec8-e1aa167ef874\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:32:04.875680 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.875637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:04.878126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.878102    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-047453-serving-cert\""
Sep 11 18:32:04.890998 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.890976    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"]
Sep 11 18:32:04.954315 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.954222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq" event={"ID":"0deda4de-c01f-4161-bdbe-33dcafa6e04d","Type":"ContainerStarted","Data":"0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751"}
Sep 11 18:32:04.954315 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.954281    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq" event={"ID":"0deda4de-c01f-4161-bdbe-33dcafa6e04d","Type":"ContainerStarted","Data":"e2389e455f10faa23d12a78a8f788549513565fc0d4652ddd5f47bbbd9595c2c"}
Sep 11 18:32:04.954516 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.954468    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:32:04.956318 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.956289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp" event={"ID":"b3980cc4-62a8-4e38-a426-870184f86296","Type":"ContainerStarted","Data":"81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f"}
Sep 11 18:32:04.956445 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.956336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp" event={"ID":"b3980cc4-62a8-4e38-a426-870184f86296","Type":"ContainerStarted","Data":"62dfa91ff03b974daeb0861fa05bcadff6dd86302c33a51c893cef67b84df1ff"}
Sep 11 18:32:04.956445 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.956441    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:32:04.960752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.960732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmwb7\" (UniqueName: \"kubernetes.io/projected/1053b8ae-1501-46f8-8b2f-b0a3488bd261-kube-api-access-vmwb7\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:04.960856 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.960762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-metrics-certs\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:04.960856 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.960836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1053b8ae-1501-46f8-8b2f-b0a3488bd261-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:04.960856 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.960853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-maas-api-tls\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:04.972113 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.972067    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq" podStartSLOduration=1.97205009 podStartE2EDuration="1.97205009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:04.970889774 +0000 UTC m=+989.550154335" watchObservedRunningTime="2026-09-11 18:32:04.97205009 +0000 UTC m=+989.551314651"
Sep 11 18:32:04.988327 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.988288    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp" podStartSLOduration=1.988275331 podStartE2EDuration="1.988275331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:04.987717363 +0000 UTC m=+989.566981927" watchObservedRunningTime="2026-09-11 18:32:04.988275331 +0000 UTC m=+989.567539891"
Sep 11 18:32:04.990797 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:04.990771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:32:05.061548 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.061521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1053b8ae-1501-46f8-8b2f-b0a3488bd261-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:05.061701 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.061557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-maas-api-tls\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:05.061701 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:32:05.061681    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-serving-cert: secret "maas-api-e2e-worker-w0-047453-serving-cert" not found
Sep 11 18:32:05.061834 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:32:05.061752    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-maas-api-tls podName:1053b8ae-1501-46f8-8b2f-b0a3488bd261 nodeName:}" failed. No retries permitted until 2026-09-11 18:32:05.561732034 +0000 UTC m=+990.140996585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-maas-api-tls") pod "maas-api-e2e-worker-w0-047453-5949858f74-qb8bq" (UID: "1053b8ae-1501-46f8-8b2f-b0a3488bd261") : secret "maas-api-e2e-worker-w0-047453-serving-cert" not found
Sep 11 18:32:05.061834 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.061820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmwb7\" (UniqueName: \"kubernetes.io/projected/1053b8ae-1501-46f8-8b2f-b0a3488bd261-kube-api-access-vmwb7\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:05.061956 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.061857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-metrics-certs\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:05.062297 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.062257    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1053b8ae-1501-46f8-8b2f-b0a3488bd261-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:05.064444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.064424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-metrics-certs\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:05.071284 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.071238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmwb7\" (UniqueName: \"kubernetes.io/projected/1053b8ae-1501-46f8-8b2f-b0a3488bd261-kube-api-access-vmwb7\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:05.260413 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.260389    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"]
Sep 11 18:32:05.262517 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:32:05.262477    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2fa513_81f4_46f1_aec8_e1aa167ef874.slice/crio-1dd454feb3a97d747c8091383be9402f940ba6effe621abb03b773015bc6943e WatchSource:0}: Error finding container 1dd454feb3a97d747c8091383be9402f940ba6effe621abb03b773015bc6943e: Status 404 returned error can't find the container with id 1dd454feb3a97d747c8091383be9402f940ba6effe621abb03b773015bc6943e
Sep 11 18:32:05.566463 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.566427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-maas-api-tls\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:05.568944 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.568912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-maas-api-tls\") pod \"maas-api-e2e-worker-w0-047453-5949858f74-qb8bq\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:05.794238 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.794201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:05.962645 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.962603    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq" event={"ID":"2b2fa513-81f4-46f1-aec8-e1aa167ef874","Type":"ContainerStarted","Data":"ac1414881c8388c4eb383d6c7f481e47c8bd7e691be375ad0e71a50fc9a41f21"}
Sep 11 18:32:05.962838 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.962688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq" event={"ID":"2b2fa513-81f4-46f1-aec8-e1aa167ef874","Type":"ContainerStarted","Data":"1dd454feb3a97d747c8091383be9402f940ba6effe621abb03b773015bc6943e"}
Sep 11 18:32:05.989212 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:05.989168    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq" podStartSLOduration=1.989154162 podStartE2EDuration="1.989154162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:05.988910113 +0000 UTC m=+990.568174675" watchObservedRunningTime="2026-09-11 18:32:05.989154162 +0000 UTC m=+990.568418721"
Sep 11 18:32:06.072438 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:06.070448    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"]
Sep 11 18:32:06.968444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:06.968406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq" event={"ID":"1053b8ae-1501-46f8-8b2f-b0a3488bd261","Type":"ContainerStarted","Data":"d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1"}
Sep 11 18:32:06.968884 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:06.968466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq" event={"ID":"1053b8ae-1501-46f8-8b2f-b0a3488bd261","Type":"ContainerStarted","Data":"b7f9e7c1c7e8285807b5c0236c23d39369d2a01ed9a2ceee5abcd6df35d11b47"}
Sep 11 18:32:06.968884 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:06.968578    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:32:06.968884 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:06.968635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:07.022321 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:07.022266    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq" podStartSLOduration=2.690895889 podStartE2EDuration="3.022253641s" totalImagesPullingTime="331.357752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:07.020867662 +0000 UTC m=+991.600132288" watchObservedRunningTime="2026-09-11 18:32:07.022253641 +0000 UTC m=+991.601518201"
Sep 11 18:32:09.962026 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:09.961995    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:32:12.978040 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:12.978010    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:32:14.399103 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.399063    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"]
Sep 11 18:32:14.402268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.402248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:14.405121 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.405083    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d6df0e9a\""
Sep 11 18:32:14.405248 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.405198    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d6df0e9a-dockercfg-4jnzz\""
Sep 11 18:32:14.412341 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.412319    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"]
Sep 11 18:32:14.452182 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.452146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9n2p\" (UniqueName: \"kubernetes.io/projected/ec7e370a-4926-4f2e-bcab-1cee76258f53-kube-api-access-h9n2p\") pod \"payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45\" (UID: \"ec7e370a-4926-4f2e-bcab-1cee76258f53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:14.452353 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.452209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7e370a-4926-4f2e-bcab-1cee76258f53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45\" (UID: \"ec7e370a-4926-4f2e-bcab-1cee76258f53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:14.552866 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.552832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9n2p\" (UniqueName: \"kubernetes.io/projected/ec7e370a-4926-4f2e-bcab-1cee76258f53-kube-api-access-h9n2p\") pod \"payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45\" (UID: \"ec7e370a-4926-4f2e-bcab-1cee76258f53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:14.553027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.552892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7e370a-4926-4f2e-bcab-1cee76258f53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45\" (UID: \"ec7e370a-4926-4f2e-bcab-1cee76258f53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:14.553798 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.553781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7e370a-4926-4f2e-bcab-1cee76258f53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45\" (UID: \"ec7e370a-4926-4f2e-bcab-1cee76258f53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:14.561701 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.561677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9n2p\" (UniqueName: \"kubernetes.io/projected/ec7e370a-4926-4f2e-bcab-1cee76258f53-kube-api-access-h9n2p\") pod \"payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45\" (UID: \"ec7e370a-4926-4f2e-bcab-1cee76258f53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:14.713533 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.713449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:14.998599 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:14.998575    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"]
Sep 11 18:32:15.000659 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:32:15.000612    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7e370a_4926_4f2e_bcab_1cee76258f53.slice/crio-f1a693d9dd2c39507b077773d1edba946375350939d4875b778c78a542ffec1e WatchSource:0}: Error finding container f1a693d9dd2c39507b077773d1edba946375350939d4875b778c78a542ffec1e: Status 404 returned error can't find the container with id f1a693d9dd2c39507b077773d1edba946375350939d4875b778c78a542ffec1e
Sep 11 18:32:15.614988 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.614955    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"]
Sep 11 18:32:15.617260 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.617244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.619742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.619721    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-536a4d-serving-cert\""
Sep 11 18:32:15.627035 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.627012    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"]
Sep 11 18:32:15.763880 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.763844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-maas-api-tls\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.764058 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.763908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-openshift-service-ca\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.764058 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.763941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2q26\" (UniqueName: \"kubernetes.io/projected/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-kube-api-access-c2q26\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.764058 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.763972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-metrics-certs\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.864863 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.864826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-maas-api-tls\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.865100 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.864892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-openshift-service-ca\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.865100 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.864922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2q26\" (UniqueName: \"kubernetes.io/projected/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-kube-api-access-c2q26\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.865100 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.864952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-metrics-certs\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.865800 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.865770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-openshift-service-ca\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.867481 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.867458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-metrics-certs\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.867607 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.867580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-maas-api-tls\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.875610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.875580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2q26\" (UniqueName: \"kubernetes.io/projected/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-kube-api-access-c2q26\") pod \"maas-api-e2e-autores-536a4d-7b88664757-89pc8\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.928547 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.928508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:15.964868 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.964838    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:32:15.965011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:15.964967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:32:16.016412 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.016378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45" event={"ID":"ec7e370a-4926-4f2e-bcab-1cee76258f53","Type":"ContainerStarted","Data":"ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed"}
Sep 11 18:32:16.016539 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.016466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45" event={"ID":"ec7e370a-4926-4f2e-bcab-1cee76258f53","Type":"ContainerStarted","Data":"f1a693d9dd2c39507b077773d1edba946375350939d4875b778c78a542ffec1e"}
Sep 11 18:32:16.016631 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.016615    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:16.035933 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.035807    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45" podStartSLOduration=2.035790048 podStartE2EDuration="2.035790048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:16.033571011 +0000 UTC m=+1000.612835572" watchObservedRunningTime="2026-09-11 18:32:16.035790048 +0000 UTC m=+1000.615054619"
Sep 11 18:32:16.205296 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.205227    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"]
Sep 11 18:32:16.208960 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:32:16.208933    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e29846f_6cfa_4a0a_aa4a_bc5bf6ea5aa1.slice/crio-c71cea54067e172c8c28fffad0979376e72dd611621161055d3bd7f7f631fe54 WatchSource:0}: Error finding container c71cea54067e172c8c28fffad0979376e72dd611621161055d3bd7f7f631fe54: Status 404 returned error can't find the container with id c71cea54067e172c8c28fffad0979376e72dd611621161055d3bd7f7f631fe54
Sep 11 18:32:16.317005 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.316975    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"]
Sep 11 18:32:16.320060 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.320045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:32:16.323236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.323216    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-536a4d-dockercfg-j6r6m\""
Sep 11 18:32:16.323717 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.323700    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-536a4d\""
Sep 11 18:32:16.333844 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.333761    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"]
Sep 11 18:32:16.469276 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.469187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j272h\" (UniqueName: \"kubernetes.io/projected/6fea2460-8477-447b-9719-6de5f6bd74aa-kube-api-access-j272h\") pod \"payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl\" (UID: \"6fea2460-8477-447b-9719-6de5f6bd74aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:32:16.469276 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.469238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fea2460-8477-447b-9719-6de5f6bd74aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl\" (UID: \"6fea2460-8477-447b-9719-6de5f6bd74aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:32:16.570626 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.570602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j272h\" (UniqueName: \"kubernetes.io/projected/6fea2460-8477-447b-9719-6de5f6bd74aa-kube-api-access-j272h\") pod \"payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl\" (UID: \"6fea2460-8477-447b-9719-6de5f6bd74aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:32:16.570769 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.570641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fea2460-8477-447b-9719-6de5f6bd74aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl\" (UID: \"6fea2460-8477-447b-9719-6de5f6bd74aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:32:16.571302 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.571280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fea2460-8477-447b-9719-6de5f6bd74aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl\" (UID: \"6fea2460-8477-447b-9719-6de5f6bd74aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:32:16.580823 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.580801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j272h\" (UniqueName: \"kubernetes.io/projected/6fea2460-8477-447b-9719-6de5f6bd74aa-kube-api-access-j272h\") pod \"payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl\" (UID: \"6fea2460-8477-447b-9719-6de5f6bd74aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:32:16.630315 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.630284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:32:16.899381 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:16.899358    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"]
Sep 11 18:32:16.901710 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:32:16.901680    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fea2460_8477_447b_9719_6de5f6bd74aa.slice/crio-ff072fbd0516ecac31895852c1fbf9c36312938ba5d33becd6dd726259f68d83 WatchSource:0}: Error finding container ff072fbd0516ecac31895852c1fbf9c36312938ba5d33becd6dd726259f68d83: Status 404 returned error can't find the container with id ff072fbd0516ecac31895852c1fbf9c36312938ba5d33becd6dd726259f68d83
Sep 11 18:32:17.022855 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:17.022822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl" event={"ID":"6fea2460-8477-447b-9719-6de5f6bd74aa","Type":"ContainerStarted","Data":"4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7"}
Sep 11 18:32:17.023040 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:17.022882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl" event={"ID":"6fea2460-8477-447b-9719-6de5f6bd74aa","Type":"ContainerStarted","Data":"ff072fbd0516ecac31895852c1fbf9c36312938ba5d33becd6dd726259f68d83"}
Sep 11 18:32:17.023040 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:17.022970    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:32:17.024417 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:17.024391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8" event={"ID":"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1","Type":"ContainerStarted","Data":"6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec"}
Sep 11 18:32:17.024539 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:17.024434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8" event={"ID":"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1","Type":"ContainerStarted","Data":"c71cea54067e172c8c28fffad0979376e72dd611621161055d3bd7f7f631fe54"}
Sep 11 18:32:17.024607 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:17.024538    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:17.045957 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:17.045913    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl" podStartSLOduration=1.04589815 podStartE2EDuration="1.04589815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:17.044029179 +0000 UTC m=+1001.623293742" watchObservedRunningTime="2026-09-11 18:32:17.04589815 +0000 UTC m=+1001.625162710"
Sep 11 18:32:17.974762 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:17.974730    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:32:17.996521 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:17.996469    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8" podStartSLOduration=2.6678240989999997 podStartE2EDuration="2.996452294s" totalImagesPullingTime="328.628195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:17.11878841 +0000 UTC m=+1001.698052969" watchObservedRunningTime="2026-09-11 18:32:17.996452294 +0000 UTC m=+1002.575716906"
Sep 11 18:32:23.037003 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:23.036973    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:32:24.724910 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.724878    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"]
Sep 11 18:32:24.727524 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.727509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:32:24.730516 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.730490    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e8f201\""
Sep 11 18:32:24.730735 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.730720    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e8f201-dockercfg-5wgv6\""
Sep 11 18:32:24.739273 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.739252    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"]
Sep 11 18:32:24.848161 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.848125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlbd\" (UniqueName: \"kubernetes.io/projected/cf238a2d-f761-4475-9397-51f024a57658-kube-api-access-5nlbd\") pod \"payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4\" (UID: \"cf238a2d-f761-4475-9397-51f024a57658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:32:24.848307 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.848246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf238a2d-f761-4475-9397-51f024a57658-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4\" (UID: \"cf238a2d-f761-4475-9397-51f024a57658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:32:24.949198 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.949168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf238a2d-f761-4475-9397-51f024a57658-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4\" (UID: \"cf238a2d-f761-4475-9397-51f024a57658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:32:24.949344 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.949253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlbd\" (UniqueName: \"kubernetes.io/projected/cf238a2d-f761-4475-9397-51f024a57658-kube-api-access-5nlbd\") pod \"payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4\" (UID: \"cf238a2d-f761-4475-9397-51f024a57658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:32:24.949973 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.949955    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf238a2d-f761-4475-9397-51f024a57658-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4\" (UID: \"cf238a2d-f761-4475-9397-51f024a57658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:32:24.957672 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:24.957620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlbd\" (UniqueName: \"kubernetes.io/projected/cf238a2d-f761-4475-9397-51f024a57658-kube-api-access-5nlbd\") pod \"payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4\" (UID: \"cf238a2d-f761-4475-9397-51f024a57658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:32:25.038215 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:25.038197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:32:25.306844 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:25.306819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"]
Sep 11 18:32:25.308198 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:32:25.308170    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf238a2d_f761_4475_9397_51f024a57658.slice/crio-742fae16f1f2ba6f0fa71afa9ad4305b7034f7dd375012d616d7720206a12667 WatchSource:0}: Error finding container 742fae16f1f2ba6f0fa71afa9ad4305b7034f7dd375012d616d7720206a12667: Status 404 returned error can't find the container with id 742fae16f1f2ba6f0fa71afa9ad4305b7034f7dd375012d616d7720206a12667
Sep 11 18:32:26.068886 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:26.068854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4" event={"ID":"cf238a2d-f761-4475-9397-51f024a57658","Type":"ContainerStarted","Data":"131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a"}
Sep 11 18:32:26.069328 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:26.068906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4" event={"ID":"cf238a2d-f761-4475-9397-51f024a57658","Type":"ContainerStarted","Data":"742fae16f1f2ba6f0fa71afa9ad4305b7034f7dd375012d616d7720206a12667"}
Sep 11 18:32:26.069328 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:26.068928    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:32:26.085777 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:26.085732    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4" podStartSLOduration=2.085717787 podStartE2EDuration="2.085717787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:26.085262255 +0000 UTC m=+1010.664526825" watchObservedRunningTime="2026-09-11 18:32:26.085717787 +0000 UTC m=+1010.664982347"
Sep 11 18:32:27.026227 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:27.026198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:28.032563 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:28.032534    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:32:37.075771 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:37.075739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:32:51.865928 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:51.865888    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"]
Sep 11 18:32:51.866373 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:51.866186    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45" podUID="ec7e370a-4926-4f2e-bcab-1cee76258f53" containerName="payload-pre-processing" containerID="cri-o://ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed" gracePeriod=30
Sep 11 18:32:52.116482 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.116427    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:52.177831 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.177802    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7e370a-4926-4f2e-bcab-1cee76258f53-plugins-config-volume\") pod \"ec7e370a-4926-4f2e-bcab-1cee76258f53\" (UID: \"ec7e370a-4926-4f2e-bcab-1cee76258f53\") "
Sep 11 18:32:52.178005 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.177971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9n2p\" (UniqueName: \"kubernetes.io/projected/ec7e370a-4926-4f2e-bcab-1cee76258f53-kube-api-access-h9n2p\") pod \"ec7e370a-4926-4f2e-bcab-1cee76258f53\" (UID: \"ec7e370a-4926-4f2e-bcab-1cee76258f53\") "
Sep 11 18:32:52.178155 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.178130    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7e370a-4926-4f2e-bcab-1cee76258f53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec7e370a-4926-4f2e-bcab-1cee76258f53" (UID: "ec7e370a-4926-4f2e-bcab-1cee76258f53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:32:52.178227 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.178208    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7e370a-4926-4f2e-bcab-1cee76258f53-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:32:52.180038 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.180018    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7e370a-4926-4f2e-bcab-1cee76258f53-kube-api-access-h9n2p" (OuterVolumeSpecName: "kube-api-access-h9n2p") pod "ec7e370a-4926-4f2e-bcab-1cee76258f53" (UID: "ec7e370a-4926-4f2e-bcab-1cee76258f53"). InnerVolumeSpecName "kube-api-access-h9n2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:32:52.198309 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.198283    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec7e370a-4926-4f2e-bcab-1cee76258f53" containerID="ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed" exitCode=0
Sep 11 18:32:52.198414 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.198331    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"
Sep 11 18:32:52.198414 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.198360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45" event={"ID":"ec7e370a-4926-4f2e-bcab-1cee76258f53","Type":"ContainerDied","Data":"ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed"}
Sep 11 18:32:52.198414 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.198412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45" event={"ID":"ec7e370a-4926-4f2e-bcab-1cee76258f53","Type":"ContainerDied","Data":"f1a693d9dd2c39507b077773d1edba946375350939d4875b778c78a542ffec1e"}
Sep 11 18:32:52.198578 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.198442    2578 scope.go:117] "RemoveContainer" containerID="ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed"
Sep 11 18:32:52.207251 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.207232    2578 scope.go:117] "RemoveContainer" containerID="ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed"
Sep 11 18:32:52.207483 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:32:52.207466    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed\": container with ID starting with ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed not found: ID does not exist" containerID="ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed"
Sep 11 18:32:52.207526 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.207490    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed"} err="failed to get container status \"ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed\": rpc error: code = NotFound desc = could not find container \"ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed\": container with ID starting with ab3f21b7ba6bc0c7496f0ba4e90064e73b4f3f3020e4e5d1e28ca5f2e6553fed not found: ID does not exist"
Sep 11 18:32:52.220433 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.220410    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"]
Sep 11 18:32:52.224587 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.224568    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6df0e9a-698bfc48cd-98x45"]
Sep 11 18:32:52.278985 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:52.278961    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9n2p\" (UniqueName: \"kubernetes.io/projected/ec7e370a-4926-4f2e-bcab-1cee76258f53-kube-api-access-h9n2p\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:32:53.937894 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:32:53.937858    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec7e370a-4926-4f2e-bcab-1cee76258f53" path="/var/lib/kubelet/pods/ec7e370a-4926-4f2e-bcab-1cee76258f53/volumes"
Sep 11 18:33:16.424380 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.424346    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"]
Sep 11 18:33:16.424805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.424786    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec7e370a-4926-4f2e-bcab-1cee76258f53" containerName="payload-pre-processing"
Sep 11 18:33:16.424805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.424800    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec7e370a-4926-4f2e-bcab-1cee76258f53" containerName="payload-pre-processing"
Sep 11 18:33:16.424881 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.424861    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec7e370a-4926-4f2e-bcab-1cee76258f53" containerName="payload-pre-processing"
Sep 11 18:33:16.428015 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.428001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.431769 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.431745    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-200c07\"/\"default-dockercfg-8cg86\""
Sep 11 18:33:16.431891 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.431791    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-200c07\"/\"kube-root-ca.crt\""
Sep 11 18:33:16.431891 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.431827    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-200c07\"/\"openshift-service-ca.crt\""
Sep 11 18:33:16.432021 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.431885    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-200c07\"/\"simulator-w4-200c07-kserve-self-signed-certs\""
Sep 11 18:33:16.436305 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.436285    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"]
Sep 11 18:33:16.599608 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.599575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tls-certs\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.599788 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.599612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-home\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.599788 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.599640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-dshm\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.599788 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.599675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-model-cache\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.599941 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.599830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tmp-dir\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.599941 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.599873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6grn\" (UniqueName: \"kubernetes.io/projected/d8c5cb61-104b-438a-858c-0b3a51eac6dc-kube-api-access-l6grn\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.701059 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.700984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tls-certs\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.701059 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.701025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-home\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.701059 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.701043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-dshm\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.701059 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.701060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-model-cache\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.701300 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.701104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tmp-dir\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.701300 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.701124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6grn\" (UniqueName: \"kubernetes.io/projected/d8c5cb61-104b-438a-858c-0b3a51eac6dc-kube-api-access-l6grn\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.701442 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.701418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-home\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.701513 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.701458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-model-cache\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.701569 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.701542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tmp-dir\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.703297 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.703269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-dshm\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.703522 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.703503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tls-certs\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.709154 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.709132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6grn\" (UniqueName: \"kubernetes.io/projected/d8c5cb61-104b-438a-858c-0b3a51eac6dc-kube-api-access-l6grn\") pod \"simulator-w4-200c07-kserve-56847d8f59-fj2lk\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") " pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:16.737167 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:16.737145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:17.009373 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:17.009345    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"]
Sep 11 18:33:17.011349 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:33:17.011324    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8c5cb61_104b_438a_858c_0b3a51eac6dc.slice/crio-1ca014899a46352135eadd26054f9e4fb6968e42319bb0f18872eafc3e221b4c WatchSource:0}: Error finding container 1ca014899a46352135eadd26054f9e4fb6968e42319bb0f18872eafc3e221b4c: Status 404 returned error can't find the container with id 1ca014899a46352135eadd26054f9e4fb6968e42319bb0f18872eafc3e221b4c
Sep 11 18:33:17.320053 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:17.320015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk" event={"ID":"d8c5cb61-104b-438a-858c-0b3a51eac6dc","Type":"ContainerStarted","Data":"c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b"}
Sep 11 18:33:17.320219 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:17.320061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk" event={"ID":"d8c5cb61-104b-438a-858c-0b3a51eac6dc","Type":"ContainerStarted","Data":"1ca014899a46352135eadd26054f9e4fb6968e42319bb0f18872eafc3e221b4c"}
Sep 11 18:33:17.338095 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:17.338044    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk" podStartSLOduration=1.33802765 podStartE2EDuration="1.33802765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:17.335846231 +0000 UTC m=+1061.915110792" watchObservedRunningTime="2026-09-11 18:33:17.33802765 +0000 UTC m=+1061.917292209"
Sep 11 18:33:26.738184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:26.738149    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:26.738184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:26.738191    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:26.750456 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:26.750428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:27.380849 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:27.380819    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:33:32.778477 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.778366    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"]
Sep 11 18:33:32.782630 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.782610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:33:32.785789 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.785764    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f8e161a6\""
Sep 11 18:33:32.785901 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.785778    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f8e161a6-dockercfg-vd62b\""
Sep 11 18:33:32.793486 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.793456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"]
Sep 11 18:33:32.847589 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.847556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9vqn\" (UniqueName: \"kubernetes.io/projected/469f96c5-6120-49e4-9f9a-c67f1b902181-kube-api-access-v9vqn\") pod \"payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv\" (UID: \"469f96c5-6120-49e4-9f9a-c67f1b902181\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:33:32.847795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.847705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/469f96c5-6120-49e4-9f9a-c67f1b902181-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv\" (UID: \"469f96c5-6120-49e4-9f9a-c67f1b902181\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:33:32.949119 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.949074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/469f96c5-6120-49e4-9f9a-c67f1b902181-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv\" (UID: \"469f96c5-6120-49e4-9f9a-c67f1b902181\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:33:32.949317 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.949142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vqn\" (UniqueName: \"kubernetes.io/projected/469f96c5-6120-49e4-9f9a-c67f1b902181-kube-api-access-v9vqn\") pod \"payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv\" (UID: \"469f96c5-6120-49e4-9f9a-c67f1b902181\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:33:32.949915 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.949890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/469f96c5-6120-49e4-9f9a-c67f1b902181-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv\" (UID: \"469f96c5-6120-49e4-9f9a-c67f1b902181\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:33:32.957531 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:32.957504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9vqn\" (UniqueName: \"kubernetes.io/projected/469f96c5-6120-49e4-9f9a-c67f1b902181-kube-api-access-v9vqn\") pod \"payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv\" (UID: \"469f96c5-6120-49e4-9f9a-c67f1b902181\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:33:33.094789 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:33.094691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:33:33.391876 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:33.391849    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"]
Sep 11 18:33:33.393949 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:33:33.393917    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469f96c5_6120_49e4_9f9a_c67f1b902181.slice/crio-4ccc968f3c0fa9a47350457fa878fa9f8737b4b14a03abbf751832a0bc7ba8bd WatchSource:0}: Error finding container 4ccc968f3c0fa9a47350457fa878fa9f8737b4b14a03abbf751832a0bc7ba8bd: Status 404 returned error can't find the container with id 4ccc968f3c0fa9a47350457fa878fa9f8737b4b14a03abbf751832a0bc7ba8bd
Sep 11 18:33:33.401509 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:33.401481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv" event={"ID":"469f96c5-6120-49e4-9f9a-c67f1b902181","Type":"ContainerStarted","Data":"4ccc968f3c0fa9a47350457fa878fa9f8737b4b14a03abbf751832a0bc7ba8bd"}
Sep 11 18:33:34.407927 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:34.407889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv" event={"ID":"469f96c5-6120-49e4-9f9a-c67f1b902181","Type":"ContainerStarted","Data":"9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d"}
Sep 11 18:33:34.408279 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:34.408009    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:33:34.427031 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:34.426979    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv" podStartSLOduration=2.426963643 podStartE2EDuration="2.426963643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:34.424288325 +0000 UTC m=+1079.003552886" watchObservedRunningTime="2026-09-11 18:33:34.426963643 +0000 UTC m=+1079.006228197"
Sep 11 18:33:45.415827 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:33:45.415797    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:34:02.541958 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:02.541919    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"]
Sep 11 18:34:02.542425 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:02.542206    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv" podUID="469f96c5-6120-49e4-9f9a-c67f1b902181" containerName="payload-processing" containerID="cri-o://9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d" gracePeriod=30
Sep 11 18:34:02.795859 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:02.795799    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:34:02.930465 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:02.930428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9vqn\" (UniqueName: \"kubernetes.io/projected/469f96c5-6120-49e4-9f9a-c67f1b902181-kube-api-access-v9vqn\") pod \"469f96c5-6120-49e4-9f9a-c67f1b902181\" (UID: \"469f96c5-6120-49e4-9f9a-c67f1b902181\") "
Sep 11 18:34:02.930660 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:02.930554    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/469f96c5-6120-49e4-9f9a-c67f1b902181-plugins-config-volume\") pod \"469f96c5-6120-49e4-9f9a-c67f1b902181\" (UID: \"469f96c5-6120-49e4-9f9a-c67f1b902181\") "
Sep 11 18:34:02.930935 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:02.930909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469f96c5-6120-49e4-9f9a-c67f1b902181-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "469f96c5-6120-49e4-9f9a-c67f1b902181" (UID: "469f96c5-6120-49e4-9f9a-c67f1b902181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:34:02.932466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:02.932442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469f96c5-6120-49e4-9f9a-c67f1b902181-kube-api-access-v9vqn" (OuterVolumeSpecName: "kube-api-access-v9vqn") pod "469f96c5-6120-49e4-9f9a-c67f1b902181" (UID: "469f96c5-6120-49e4-9f9a-c67f1b902181"). InnerVolumeSpecName "kube-api-access-v9vqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:34:03.031378 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.031339    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/469f96c5-6120-49e4-9f9a-c67f1b902181-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:34:03.031378 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.031370    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9vqn\" (UniqueName: \"kubernetes.io/projected/469f96c5-6120-49e4-9f9a-c67f1b902181-kube-api-access-v9vqn\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:34:03.555031 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.554997    2578 generic.go:356] "Generic (PLEG): container finished" podID="469f96c5-6120-49e4-9f9a-c67f1b902181" containerID="9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d" exitCode=0
Sep 11 18:34:03.555527 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.555045    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"
Sep 11 18:34:03.555527 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.555082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv" event={"ID":"469f96c5-6120-49e4-9f9a-c67f1b902181","Type":"ContainerDied","Data":"9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d"}
Sep 11 18:34:03.555527 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.555141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv" event={"ID":"469f96c5-6120-49e4-9f9a-c67f1b902181","Type":"ContainerDied","Data":"4ccc968f3c0fa9a47350457fa878fa9f8737b4b14a03abbf751832a0bc7ba8bd"}
Sep 11 18:34:03.555527 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.555163    2578 scope.go:117] "RemoveContainer" containerID="9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d"
Sep 11 18:34:03.565795 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.565767    2578 scope.go:117] "RemoveContainer" containerID="9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d"
Sep 11 18:34:03.566578 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:34:03.566553    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d\": container with ID starting with 9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d not found: ID does not exist" containerID="9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d"
Sep 11 18:34:03.566707 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.566585    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d"} err="failed to get container status \"9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d\": rpc error: code = NotFound desc = could not find container \"9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d\": container with ID starting with 9fe5c44c17a5898fd574f3fa4373e918b0aa7ba6d599d7b75999420eff43171d not found: ID does not exist"
Sep 11 18:34:03.578402 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.578373    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"]
Sep 11 18:34:03.583733 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.583710    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e161a6-779c7fccff-4z5hv"]
Sep 11 18:34:03.937851 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:03.937774    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="469f96c5-6120-49e4-9f9a-c67f1b902181" path="/var/lib/kubelet/pods/469f96c5-6120-49e4-9f9a-c67f1b902181/volumes"
Sep 11 18:34:09.093170 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.093134    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"]
Sep 11 18:34:09.093549 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.093536    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469f96c5-6120-49e4-9f9a-c67f1b902181" containerName="payload-processing"
Sep 11 18:34:09.093549 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.093550    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="469f96c5-6120-49e4-9f9a-c67f1b902181" containerName="payload-processing"
Sep 11 18:34:09.093630 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.093610    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="469f96c5-6120-49e4-9f9a-c67f1b902181" containerName="payload-processing"
Sep 11 18:34:09.098138 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.098118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:09.101078 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.101057    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0b2b4b09\""
Sep 11 18:34:09.112302 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.112270    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"]
Sep 11 18:34:09.190336 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.190302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4bcc24-1cbb-4b63-bafc-3350858d3339-plugins-config-volume\") pod \"payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h\" (UID: \"ec4bcc24-1cbb-4b63-bafc-3350858d3339\") " pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:09.190336 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.190338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cjr\" (UniqueName: \"kubernetes.io/projected/ec4bcc24-1cbb-4b63-bafc-3350858d3339-kube-api-access-g4cjr\") pod \"payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h\" (UID: \"ec4bcc24-1cbb-4b63-bafc-3350858d3339\") " pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:09.291703 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.291667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4bcc24-1cbb-4b63-bafc-3350858d3339-plugins-config-volume\") pod \"payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h\" (UID: \"ec4bcc24-1cbb-4b63-bafc-3350858d3339\") " pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:09.291877 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.291716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cjr\" (UniqueName: \"kubernetes.io/projected/ec4bcc24-1cbb-4b63-bafc-3350858d3339-kube-api-access-g4cjr\") pod \"payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h\" (UID: \"ec4bcc24-1cbb-4b63-bafc-3350858d3339\") " pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:09.292439 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.292419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4bcc24-1cbb-4b63-bafc-3350858d3339-plugins-config-volume\") pod \"payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h\" (UID: \"ec4bcc24-1cbb-4b63-bafc-3350858d3339\") " pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:09.303450 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.303416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cjr\" (UniqueName: \"kubernetes.io/projected/ec4bcc24-1cbb-4b63-bafc-3350858d3339-kube-api-access-g4cjr\") pod \"payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h\" (UID: \"ec4bcc24-1cbb-4b63-bafc-3350858d3339\") " pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:09.408320 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.408229    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:09.682115 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:09.682085    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"]
Sep 11 18:34:09.684137 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:34:09.684108    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4bcc24_1cbb_4b63_bafc_3350858d3339.slice/crio-08893874012fd36473d0ccebc73347c3a151356fd0cf2f5afae554dd758148cf WatchSource:0}: Error finding container 08893874012fd36473d0ccebc73347c3a151356fd0cf2f5afae554dd758148cf: Status 404 returned error can't find the container with id 08893874012fd36473d0ccebc73347c3a151356fd0cf2f5afae554dd758148cf
Sep 11 18:34:10.593614 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:10.593578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h" event={"ID":"ec4bcc24-1cbb-4b63-bafc-3350858d3339","Type":"ContainerStarted","Data":"cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71"}
Sep 11 18:34:10.594096 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:10.593630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h" event={"ID":"ec4bcc24-1cbb-4b63-bafc-3350858d3339","Type":"ContainerStarted","Data":"08893874012fd36473d0ccebc73347c3a151356fd0cf2f5afae554dd758148cf"}
Sep 11 18:34:10.594096 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:10.593678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:15.332881 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.332824    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h" podStartSLOduration=6.332806258 podStartE2EDuration="6.332806258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:10.623731355 +0000 UTC m=+1115.202995916" watchObservedRunningTime="2026-09-11 18:34:15.332806258 +0000 UTC m=+1119.912070818"
Sep 11 18:34:15.334348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.334325    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"]
Sep 11 18:34:15.338024 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.338005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.342006 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.341984    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-536a4d\"/\"autores-model-536a4d-kserve-self-signed-certs\""
Sep 11 18:34:15.342006 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.341988    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-536a4d\"/\"default-dockercfg-kljs2\""
Sep 11 18:34:15.342190 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.342033    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-536a4d\"/\"kube-root-ca.crt\""
Sep 11 18:34:15.342190 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.341985    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-536a4d\"/\"openshift-service-ca.crt\""
Sep 11 18:34:15.347515 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.347495    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"]
Sep 11 18:34:15.446269 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.446238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-tmp-dir\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.446482 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.446276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dncnf\" (UniqueName: \"kubernetes.io/projected/07a9f702-a469-487e-8bb6-ade55b142ee9-kube-api-access-dncnf\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.446482 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.446299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-home\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.446482 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.446362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07a9f702-a469-487e-8bb6-ade55b142ee9-tls-certs\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.446482 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.446405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-dshm\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.446482 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.446438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-model-cache\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.547198 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.547164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07a9f702-a469-487e-8bb6-ade55b142ee9-tls-certs\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.547393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.547226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-dshm\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.547393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.547255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-model-cache\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.547393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.547275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-tmp-dir\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.547393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.547296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dncnf\" (UniqueName: \"kubernetes.io/projected/07a9f702-a469-487e-8bb6-ade55b142ee9-kube-api-access-dncnf\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.547393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.547317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-home\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.547794 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.547757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-tmp-dir\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.547919 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.547829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-home\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.547919 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.547853    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-model-cache\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.549562 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.549542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-dshm\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.549823 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.549808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07a9f702-a469-487e-8bb6-ade55b142ee9-tls-certs\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.556803 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.556781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dncnf\" (UniqueName: \"kubernetes.io/projected/07a9f702-a469-487e-8bb6-ade55b142ee9-kube-api-access-dncnf\") pod \"autores-model-536a4d-kserve-5c8965d49f-mc9kr\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") " pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.648025 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.647948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:15.921321 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:15.921294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"]
Sep 11 18:34:15.923338 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:34:15.923309    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07a9f702_a469_487e_8bb6_ade55b142ee9.slice/crio-69577c6fa3bbe904e157cb011e894c3d6e7ab5bb46c1eaf44724c10e95f08d25 WatchSource:0}: Error finding container 69577c6fa3bbe904e157cb011e894c3d6e7ab5bb46c1eaf44724c10e95f08d25: Status 404 returned error can't find the container with id 69577c6fa3bbe904e157cb011e894c3d6e7ab5bb46c1eaf44724c10e95f08d25
Sep 11 18:34:16.622847 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:16.622807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr" event={"ID":"07a9f702-a469-487e-8bb6-ade55b142ee9","Type":"ContainerStarted","Data":"d03cc050b437e18ab8a8d0de5b32ff68b1b6a0b2dffe59d49f867208f9df156a"}
Sep 11 18:34:16.623372 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:16.622858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr" event={"ID":"07a9f702-a469-487e-8bb6-ade55b142ee9","Type":"ContainerStarted","Data":"69577c6fa3bbe904e157cb011e894c3d6e7ab5bb46c1eaf44724c10e95f08d25"}
Sep 11 18:34:16.645481 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:16.645430    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr" podStartSLOduration=1.645416072 podStartE2EDuration="1.645416072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:16.644164097 +0000 UTC m=+1121.223428655" watchObservedRunningTime="2026-09-11 18:34:16.645416072 +0000 UTC m=+1121.224680632"
Sep 11 18:34:21.600379 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:21.600349    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:25.648249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:25.648206    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:25.648249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:25.648259    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:25.660714 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:25.660695    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:25.684443 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:25.684415    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:34:39.358854 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.358817    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"]
Sep 11 18:34:39.359236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.359069    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h" podUID="ec4bcc24-1cbb-4b63-bafc-3350858d3339" containerName="payload-processing" containerID="cri-o://cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71" gracePeriod=30
Sep 11 18:34:39.617970 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.617908    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:39.744696 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.744644    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec4bcc24-1cbb-4b63-bafc-3350858d3339" containerID="cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71" exitCode=0
Sep 11 18:34:39.744696 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.744680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h" event={"ID":"ec4bcc24-1cbb-4b63-bafc-3350858d3339","Type":"ContainerDied","Data":"cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71"}
Sep 11 18:34:39.745027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.744717    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"
Sep 11 18:34:39.745027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.744746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h" event={"ID":"ec4bcc24-1cbb-4b63-bafc-3350858d3339","Type":"ContainerDied","Data":"08893874012fd36473d0ccebc73347c3a151356fd0cf2f5afae554dd758148cf"}
Sep 11 18:34:39.745027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.744777    2578 scope.go:117] "RemoveContainer" containerID="cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71"
Sep 11 18:34:39.755468 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.755436    2578 scope.go:117] "RemoveContainer" containerID="cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71"
Sep 11 18:34:39.755758 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:34:39.755740    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71\": container with ID starting with cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71 not found: ID does not exist" containerID="cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71"
Sep 11 18:34:39.755815 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.755777    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71"} err="failed to get container status \"cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71\": rpc error: code = NotFound desc = could not find container \"cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71\": container with ID starting with cced56ea5b5fc381dc47466d04f6b7b69bb384d4d66206336185867fcda17c71 not found: ID does not exist"
Sep 11 18:34:39.784309 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.784285    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4bcc24-1cbb-4b63-bafc-3350858d3339-plugins-config-volume\") pod \"ec4bcc24-1cbb-4b63-bafc-3350858d3339\" (UID: \"ec4bcc24-1cbb-4b63-bafc-3350858d3339\") "
Sep 11 18:34:39.784395 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.784320    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4cjr\" (UniqueName: \"kubernetes.io/projected/ec4bcc24-1cbb-4b63-bafc-3350858d3339-kube-api-access-g4cjr\") pod \"ec4bcc24-1cbb-4b63-bafc-3350858d3339\" (UID: \"ec4bcc24-1cbb-4b63-bafc-3350858d3339\") "
Sep 11 18:34:39.784608 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.784587    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4bcc24-1cbb-4b63-bafc-3350858d3339-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec4bcc24-1cbb-4b63-bafc-3350858d3339" (UID: "ec4bcc24-1cbb-4b63-bafc-3350858d3339"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:34:39.786238 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.786213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4bcc24-1cbb-4b63-bafc-3350858d3339-kube-api-access-g4cjr" (OuterVolumeSpecName: "kube-api-access-g4cjr") pod "ec4bcc24-1cbb-4b63-bafc-3350858d3339" (UID: "ec4bcc24-1cbb-4b63-bafc-3350858d3339"). InnerVolumeSpecName "kube-api-access-g4cjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:34:39.885084 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.885017    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4bcc24-1cbb-4b63-bafc-3350858d3339-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:34:39.885084 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:39.885041    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4cjr\" (UniqueName: \"kubernetes.io/projected/ec4bcc24-1cbb-4b63-bafc-3350858d3339-kube-api-access-g4cjr\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:34:40.062612 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:40.062583    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"]
Sep 11 18:34:40.067017 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:40.066993    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b2b4b09-6d7b97fc69-h825h"]
Sep 11 18:34:41.938739 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:41.938707    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec4bcc24-1cbb-4b63-bafc-3350858d3339" path="/var/lib/kubelet/pods/ec4bcc24-1cbb-4b63-bafc-3350858d3339/volumes"
Sep 11 18:34:57.993970 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:57.993887    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"]
Sep 11 18:34:57.994410 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:57.994118    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8" podUID="2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" containerName="maas-api" containerID="cri-o://6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec" gracePeriod=30
Sep 11 18:34:58.030561 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.030528    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8" podUID="2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.42:8443/health\": dial tcp 10.133.0.42:8443: connect: connection refused"
Sep 11 18:34:58.082718 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.082687    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"]
Sep 11 18:34:58.082961 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.082929    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl" podUID="6fea2460-8477-447b-9719-6de5f6bd74aa" containerName="payload-pre-processing" containerID="cri-o://4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7" gracePeriod=30
Sep 11 18:34:58.330438 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.330405    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:34:58.347240 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.347214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-openshift-service-ca\") pod \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") "
Sep 11 18:34:58.347383 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.347280    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-metrics-certs\") pod \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") "
Sep 11 18:34:58.347383 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.347359    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-maas-api-tls\") pod \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") "
Sep 11 18:34:58.347496 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.347400    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2q26\" (UniqueName: \"kubernetes.io/projected/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-kube-api-access-c2q26\") pod \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\" (UID: \"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1\") "
Sep 11 18:34:58.347598 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.347571    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" (UID: "2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:34:58.347757 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.347739    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-openshift-service-ca\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:34:58.349484 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.349457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-kube-api-access-c2q26" (OuterVolumeSpecName: "kube-api-access-c2q26") pod "2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" (UID: "2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1"). InnerVolumeSpecName "kube-api-access-c2q26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:34:58.349568 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.349464    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" (UID: "2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:34:58.349568 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.349495    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" (UID: "2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:34:58.362399 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.362379    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:34:58.449091 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.449055    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j272h\" (UniqueName: \"kubernetes.io/projected/6fea2460-8477-447b-9719-6de5f6bd74aa-kube-api-access-j272h\") pod \"6fea2460-8477-447b-9719-6de5f6bd74aa\" (UID: \"6fea2460-8477-447b-9719-6de5f6bd74aa\") "
Sep 11 18:34:58.449273 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.449197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fea2460-8477-447b-9719-6de5f6bd74aa-plugins-config-volume\") pod \"6fea2460-8477-447b-9719-6de5f6bd74aa\" (UID: \"6fea2460-8477-447b-9719-6de5f6bd74aa\") "
Sep 11 18:34:58.449462 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.449447    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2q26\" (UniqueName: \"kubernetes.io/projected/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-kube-api-access-c2q26\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:34:58.449519 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.449466    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-metrics-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:34:58.449519 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.449480    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1-maas-api-tls\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:34:58.449590 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.449537    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fea2460-8477-447b-9719-6de5f6bd74aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fea2460-8477-447b-9719-6de5f6bd74aa" (UID: "6fea2460-8477-447b-9719-6de5f6bd74aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:34:58.451365 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.451343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fea2460-8477-447b-9719-6de5f6bd74aa-kube-api-access-j272h" (OuterVolumeSpecName: "kube-api-access-j272h") pod "6fea2460-8477-447b-9719-6de5f6bd74aa" (UID: "6fea2460-8477-447b-9719-6de5f6bd74aa"). InnerVolumeSpecName "kube-api-access-j272h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:34:58.551070 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.550989    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fea2460-8477-447b-9719-6de5f6bd74aa-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:34:58.551070 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.551027    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j272h\" (UniqueName: \"kubernetes.io/projected/6fea2460-8477-447b-9719-6de5f6bd74aa-kube-api-access-j272h\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:34:58.835981 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.835900    2578 generic.go:356] "Generic (PLEG): container finished" podID="6fea2460-8477-447b-9719-6de5f6bd74aa" containerID="4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7" exitCode=0
Sep 11 18:34:58.835981 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.835943    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"
Sep 11 18:34:58.836210 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.835988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl" event={"ID":"6fea2460-8477-447b-9719-6de5f6bd74aa","Type":"ContainerDied","Data":"4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7"}
Sep 11 18:34:58.836210 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.836048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl" event={"ID":"6fea2460-8477-447b-9719-6de5f6bd74aa","Type":"ContainerDied","Data":"ff072fbd0516ecac31895852c1fbf9c36312938ba5d33becd6dd726259f68d83"}
Sep 11 18:34:58.836210 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.836079    2578 scope.go:117] "RemoveContainer" containerID="4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7"
Sep 11 18:34:58.837400 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.837373    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" containerID="6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec" exitCode=0
Sep 11 18:34:58.837525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.837404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8" event={"ID":"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1","Type":"ContainerDied","Data":"6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec"}
Sep 11 18:34:58.837525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.837460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8" event={"ID":"2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1","Type":"ContainerDied","Data":"c71cea54067e172c8c28fffad0979376e72dd611621161055d3bd7f7f631fe54"}
Sep 11 18:34:58.837525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.837414    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"
Sep 11 18:34:58.844643 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.844626    2578 scope.go:117] "RemoveContainer" containerID="4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7"
Sep 11 18:34:58.844956 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:34:58.844937    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7\": container with ID starting with 4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7 not found: ID does not exist" containerID="4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7"
Sep 11 18:34:58.845035 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.844964    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7"} err="failed to get container status \"4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7\": rpc error: code = NotFound desc = could not find container \"4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7\": container with ID starting with 4469360f1f7445b5cee6538b7b2aa2fc670edd0c1578c051b1a0e72f0867b6c7 not found: ID does not exist"
Sep 11 18:34:58.845035 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.844986    2578 scope.go:117] "RemoveContainer" containerID="6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec"
Sep 11 18:34:58.853103 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.853087    2578 scope.go:117] "RemoveContainer" containerID="6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec"
Sep 11 18:34:58.853345 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:34:58.853327    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec\": container with ID starting with 6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec not found: ID does not exist" containerID="6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec"
Sep 11 18:34:58.853385 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.853352    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec"} err="failed to get container status \"6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec\": rpc error: code = NotFound desc = could not find container \"6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec\": container with ID starting with 6de360842da91d030b86d010cc2437175e419d186b75a95fa03675745cb6eeec not found: ID does not exist"
Sep 11 18:34:58.868475 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.868244    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"]
Sep 11 18:34:58.870291 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.870265    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-536a4d-f6cbd59d6-gsqsl"]
Sep 11 18:34:58.883356 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.883326    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"]
Sep 11 18:34:58.889153 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:58.889135    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-536a4d-7b88664757-89pc8"]
Sep 11 18:34:59.937244 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:59.937213    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" path="/var/lib/kubelet/pods/2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1/volumes"
Sep 11 18:34:59.937624 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:34:59.937609    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fea2460-8477-447b-9719-6de5f6bd74aa" path="/var/lib/kubelet/pods/6fea2460-8477-447b-9719-6de5f6bd74aa/volumes"
Sep 11 18:35:03.781719 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:03.781685    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"]
Sep 11 18:35:18.800753 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:18.800709    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr" podUID="07a9f702-a469-487e-8bb6-ade55b142ee9" containerName="main" containerID="cri-o://d03cc050b437e18ab8a8d0de5b32ff68b1b6a0b2dffe59d49f867208f9df156a" gracePeriod=45
Sep 11 18:35:18.940458 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:18.940421    2578 generic.go:356] "Generic (PLEG): container finished" podID="07a9f702-a469-487e-8bb6-ade55b142ee9" containerID="d03cc050b437e18ab8a8d0de5b32ff68b1b6a0b2dffe59d49f867208f9df156a" exitCode=0
Sep 11 18:35:18.940592 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:18.940454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr" event={"ID":"07a9f702-a469-487e-8bb6-ade55b142ee9","Type":"ContainerDied","Data":"d03cc050b437e18ab8a8d0de5b32ff68b1b6a0b2dffe59d49f867208f9df156a"}
Sep 11 18:35:19.045121 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.045096    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:35:19.149638 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.149551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-home\") pod \"07a9f702-a469-487e-8bb6-ade55b142ee9\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") "
Sep 11 18:35:19.149638 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.149600    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-tmp-dir\") pod \"07a9f702-a469-487e-8bb6-ade55b142ee9\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") "
Sep 11 18:35:19.149638 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.149630    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dncnf\" (UniqueName: \"kubernetes.io/projected/07a9f702-a469-487e-8bb6-ade55b142ee9-kube-api-access-dncnf\") pod \"07a9f702-a469-487e-8bb6-ade55b142ee9\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") "
Sep 11 18:35:19.149934 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.149677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07a9f702-a469-487e-8bb6-ade55b142ee9-tls-certs\") pod \"07a9f702-a469-487e-8bb6-ade55b142ee9\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") "
Sep 11 18:35:19.149934 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.149741    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-dshm\") pod \"07a9f702-a469-487e-8bb6-ade55b142ee9\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") "
Sep 11 18:35:19.149934 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.149814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-model-cache\") pod \"07a9f702-a469-487e-8bb6-ade55b142ee9\" (UID: \"07a9f702-a469-487e-8bb6-ade55b142ee9\") "
Sep 11 18:35:19.149934 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.149905    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07a9f702-a469-487e-8bb6-ade55b142ee9" (UID: "07a9f702-a469-487e-8bb6-ade55b142ee9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:19.150142 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.149941    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-home" (OuterVolumeSpecName: "home") pod "07a9f702-a469-487e-8bb6-ade55b142ee9" (UID: "07a9f702-a469-487e-8bb6-ade55b142ee9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:19.150142 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.150105    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-model-cache" (OuterVolumeSpecName: "model-cache") pod "07a9f702-a469-487e-8bb6-ade55b142ee9" (UID: "07a9f702-a469-487e-8bb6-ade55b142ee9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:19.150245 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.150199    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:35:19.150245 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.150226    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:35:19.150245 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.150235    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:35:19.151976 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.151949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a9f702-a469-487e-8bb6-ade55b142ee9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07a9f702-a469-487e-8bb6-ade55b142ee9" (UID: "07a9f702-a469-487e-8bb6-ade55b142ee9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:19.152070 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.151972    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-dshm" (OuterVolumeSpecName: "dshm") pod "07a9f702-a469-487e-8bb6-ade55b142ee9" (UID: "07a9f702-a469-487e-8bb6-ade55b142ee9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:19.152070 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.152039    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a9f702-a469-487e-8bb6-ade55b142ee9-kube-api-access-dncnf" (OuterVolumeSpecName: "kube-api-access-dncnf") pod "07a9f702-a469-487e-8bb6-ade55b142ee9" (UID: "07a9f702-a469-487e-8bb6-ade55b142ee9"). InnerVolumeSpecName "kube-api-access-dncnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:19.251617 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.251580    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dncnf\" (UniqueName: \"kubernetes.io/projected/07a9f702-a469-487e-8bb6-ade55b142ee9-kube-api-access-dncnf\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:35:19.251617 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.251611    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07a9f702-a469-487e-8bb6-ade55b142ee9-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:35:19.251617 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.251621    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07a9f702-a469-487e-8bb6-ade55b142ee9-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:35:19.946637 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.946608    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"
Sep 11 18:35:19.946965 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.946675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr" event={"ID":"07a9f702-a469-487e-8bb6-ade55b142ee9","Type":"ContainerDied","Data":"69577c6fa3bbe904e157cb011e894c3d6e7ab5bb46c1eaf44724c10e95f08d25"}
Sep 11 18:35:19.946965 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.946722    2578 scope.go:117] "RemoveContainer" containerID="d03cc050b437e18ab8a8d0de5b32ff68b1b6a0b2dffe59d49f867208f9df156a"
Sep 11 18:35:19.971154 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.971123    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"]
Sep 11 18:35:19.976573 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:19.976553    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-536a4d/autores-model-536a4d-kserve-5c8965d49f-mc9kr"]
Sep 11 18:35:21.938446 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:21.938405    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07a9f702-a469-487e-8bb6-ade55b142ee9" path="/var/lib/kubelet/pods/07a9f702-a469-487e-8bb6-ade55b142ee9/volumes"
Sep 11 18:35:32.408731 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.408698    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"]
Sep 11 18:35:32.409115 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409079    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a9f702-a469-487e-8bb6-ade55b142ee9" containerName="main"
Sep 11 18:35:32.409115 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409090    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a9f702-a469-487e-8bb6-ade55b142ee9" containerName="main"
Sep 11 18:35:32.409115 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409106    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" containerName="maas-api"
Sep 11 18:35:32.409115 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409111    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" containerName="maas-api"
Sep 11 18:35:32.409115 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409117    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fea2460-8477-447b-9719-6de5f6bd74aa" containerName="payload-pre-processing"
Sep 11 18:35:32.409286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409123    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fea2460-8477-447b-9719-6de5f6bd74aa" containerName="payload-pre-processing"
Sep 11 18:35:32.409286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409132    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4bcc24-1cbb-4b63-bafc-3350858d3339" containerName="payload-processing"
Sep 11 18:35:32.409286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409137    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4bcc24-1cbb-4b63-bafc-3350858d3339" containerName="payload-processing"
Sep 11 18:35:32.409286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409189    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="07a9f702-a469-487e-8bb6-ade55b142ee9" containerName="main"
Sep 11 18:35:32.409286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409197    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fea2460-8477-447b-9719-6de5f6bd74aa" containerName="payload-pre-processing"
Sep 11 18:35:32.409286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409204    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec4bcc24-1cbb-4b63-bafc-3350858d3339" containerName="payload-processing"
Sep 11 18:35:32.409286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.409212    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e29846f-6cfa-4a0a-aa4a-bc5bf6ea5aa1" containerName="maas-api"
Sep 11 18:35:32.413949 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.413931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.417736 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.417715    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-200c07\"/\"premium-w4-200c07-kserve-self-signed-certs\""
Sep 11 18:35:32.427881 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.427859    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"]
Sep 11 18:35:32.571327 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.571300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-model-cache\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.571472 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.571341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-home\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.571472 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.571368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-tmp-dir\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.571472 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.571439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dtk\" (UniqueName: \"kubernetes.io/projected/b12ecfcc-4797-4516-b42c-175e83500be0-kube-api-access-m8dtk\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.571472 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.571465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-dshm\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.571613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.571504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12ecfcc-4797-4516-b42c-175e83500be0-tls-certs\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.672527 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.672451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dtk\" (UniqueName: \"kubernetes.io/projected/b12ecfcc-4797-4516-b42c-175e83500be0-kube-api-access-m8dtk\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.672527 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.672484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-dshm\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.672779 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.672526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12ecfcc-4797-4516-b42c-175e83500be0-tls-certs\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.672779 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.672622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-model-cache\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.672779 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.672674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-home\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.672779 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.672699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-tmp-dir\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.673101 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.673078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-model-cache\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.673101 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.673097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-home\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.673211 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.673124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-tmp-dir\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.674798 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.674778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-dshm\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.674968 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.674954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12ecfcc-4797-4516-b42c-175e83500be0-tls-certs\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.682584 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.682563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dtk\" (UniqueName: \"kubernetes.io/projected/b12ecfcc-4797-4516-b42c-175e83500be0-kube-api-access-m8dtk\") pod \"premium-w4-200c07-kserve-768cf64954-mm4tx\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") " pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.722272 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.722253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:32.999849 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:32.999670    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"]
Sep 11 18:35:33.006514 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:35:33.006485    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb12ecfcc_4797_4516_b42c_175e83500be0.slice/crio-2b375e32aaa7b5210099559238731f685a093f4dc3762d3d5c4aad377d8b2315 WatchSource:0}: Error finding container 2b375e32aaa7b5210099559238731f685a093f4dc3762d3d5c4aad377d8b2315: Status 404 returned error can't find the container with id 2b375e32aaa7b5210099559238731f685a093f4dc3762d3d5c4aad377d8b2315
Sep 11 18:35:33.014368 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:33.014344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx" event={"ID":"b12ecfcc-4797-4516-b42c-175e83500be0","Type":"ContainerStarted","Data":"2b375e32aaa7b5210099559238731f685a093f4dc3762d3d5c4aad377d8b2315"}
Sep 11 18:35:34.019883 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:34.019844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx" event={"ID":"b12ecfcc-4797-4516-b42c-175e83500be0","Type":"ContainerStarted","Data":"160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e"}
Sep 11 18:35:34.046366 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:34.046311    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx" podStartSLOduration=2.046297226 podStartE2EDuration="2.046297226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:34.045159815 +0000 UTC m=+1198.624424376" watchObservedRunningTime="2026-09-11 18:35:34.046297226 +0000 UTC m=+1198.625561784"
Sep 11 18:35:42.722949 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:42.722917    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:42.723374 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:42.723183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:42.735171 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:42.735142    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:43.068471 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:43.068438    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:35:45.768180 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.768145    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"]
Sep 11 18:35:45.772969 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.772948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.775852 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.775829    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-047453\"/\"kube-root-ca.crt\""
Sep 11 18:35:45.777027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.777007    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-047453\"/\"openshift-service-ca.crt\""
Sep 11 18:35:45.777129 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.777051    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-047453\"/\"default-dockercfg-6srvg\""
Sep 11 18:35:45.777129 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.777098    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-047453\"/\"premium-w0-047453-kserve-self-signed-certs\""
Sep 11 18:35:45.787568 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.787549    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"]
Sep 11 18:35:45.896967 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.896930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-model-cache\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.897134 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.896974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tls-certs\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.897134 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.897087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5g7h\" (UniqueName: \"kubernetes.io/projected/9dfe1397-45ce-454d-b9d1-af3c7fce823a-kube-api-access-c5g7h\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.897134 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.897115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-home\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.897263 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.897154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-dshm\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.897263 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.897242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tmp-dir\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.997993 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.997959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-model-cache\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.997993 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.997990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tls-certs\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.998235 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.998045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5g7h\" (UniqueName: \"kubernetes.io/projected/9dfe1397-45ce-454d-b9d1-af3c7fce823a-kube-api-access-c5g7h\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.998235 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.998063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-home\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.998235 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.998090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-dshm\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.998235 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.998155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tmp-dir\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.998455 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.998372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-model-cache\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.998455 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.998405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-home\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:45.998565 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:45.998486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tmp-dir\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:46.000286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:46.000262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-dshm\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:46.000587 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:46.000573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tls-certs\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:46.020037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:46.019986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5g7h\" (UniqueName: \"kubernetes.io/projected/9dfe1397-45ce-454d-b9d1-af3c7fce823a-kube-api-access-c5g7h\") pod \"premium-w0-047453-kserve-7bc46ff75c-r4z6z\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") " pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:46.082553 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:46.082521    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:46.365421 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:46.365394    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"]
Sep 11 18:35:46.367593 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:35:46.367561    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dfe1397_45ce_454d_b9d1_af3c7fce823a.slice/crio-6222fbd4e5ece9241ed60ce1452c2ae2f6c54d6f4aed4f4bbfe57c92bc9ce9df WatchSource:0}: Error finding container 6222fbd4e5ece9241ed60ce1452c2ae2f6c54d6f4aed4f4bbfe57c92bc9ce9df: Status 404 returned error can't find the container with id 6222fbd4e5ece9241ed60ce1452c2ae2f6c54d6f4aed4f4bbfe57c92bc9ce9df
Sep 11 18:35:47.079084 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:47.079044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z" event={"ID":"9dfe1397-45ce-454d-b9d1-af3c7fce823a","Type":"ContainerStarted","Data":"50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1"}
Sep 11 18:35:47.079447 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:47.079094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z" event={"ID":"9dfe1397-45ce-454d-b9d1-af3c7fce823a","Type":"ContainerStarted","Data":"6222fbd4e5ece9241ed60ce1452c2ae2f6c54d6f4aed4f4bbfe57c92bc9ce9df"}
Sep 11 18:35:47.129862 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:47.129809    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z" podStartSLOduration=2.129793411 podStartE2EDuration="2.129793411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:47.129668995 +0000 UTC m=+1211.708933552" watchObservedRunningTime="2026-09-11 18:35:47.129793411 +0000 UTC m=+1211.709057970"
Sep 11 18:35:56.083377 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:56.083338    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:56.083818 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:56.083687    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:56.095890 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:56.095862    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:35:56.132423 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:35:56.132386    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:36:14.772548 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.772508    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"]
Sep 11 18:36:14.776218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.776192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.780817 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.780790    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e8f201\"/\"kube-root-ca.crt\""
Sep 11 18:36:14.780817 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.780805    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e8f201\"/\"openshift-service-ca.crt\""
Sep 11 18:36:14.781047 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.780802    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e8f201\"/\"test-model-e8f201-kserve-self-signed-certs\""
Sep 11 18:36:14.781047 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.780934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e8f201\"/\"default-dockercfg-gg8vc\""
Sep 11 18:36:14.788197 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.788174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"]
Sep 11 18:36:14.868252 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.868209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-home\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.868252 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.868254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-model-cache\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.868449 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.868282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-dshm\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.868449 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.868346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tls-certs\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.868449 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.868393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tmp-dir\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.868449 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.868427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xj7t\" (UniqueName: \"kubernetes.io/projected/31f20dfd-06b2-4d49-8d46-38474daa3d3a-kube-api-access-7xj7t\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.969980 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.969945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-home\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.970166 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.969989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-model-cache\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.970166 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.970020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-dshm\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.970166 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.970065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tls-certs\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.970166 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.970098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tmp-dir\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.970166 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.970144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xj7t\" (UniqueName: \"kubernetes.io/projected/31f20dfd-06b2-4d49-8d46-38474daa3d3a-kube-api-access-7xj7t\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.970437 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.970401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-model-cache\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.970525 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.970501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tmp-dir\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.970605 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.970591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-home\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.972286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.972264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-dshm\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.972445 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.972429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tls-certs\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:14.979999 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:14.979974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xj7t\" (UniqueName: \"kubernetes.io/projected/31f20dfd-06b2-4d49-8d46-38474daa3d3a-kube-api-access-7xj7t\") pod \"test-model-e8f201-kserve-5c586bb95c-nvk8z\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") " pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:15.086037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:15.085950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:15.363365 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:15.363173    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"]
Sep 11 18:36:15.366299 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:36:15.366269    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31f20dfd_06b2_4d49_8d46_38474daa3d3a.slice/crio-9b587c0c09c4d19e65574783bcc21058f5bc73df7088378c74372bef572e47f3 WatchSource:0}: Error finding container 9b587c0c09c4d19e65574783bcc21058f5bc73df7088378c74372bef572e47f3: Status 404 returned error can't find the container with id 9b587c0c09c4d19e65574783bcc21058f5bc73df7088378c74372bef572e47f3
Sep 11 18:36:16.218624 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:16.218586    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z" event={"ID":"31f20dfd-06b2-4d49-8d46-38474daa3d3a","Type":"ContainerStarted","Data":"fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909"}
Sep 11 18:36:16.218624 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:16.218629    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z" event={"ID":"31f20dfd-06b2-4d49-8d46-38474daa3d3a","Type":"ContainerStarted","Data":"9b587c0c09c4d19e65574783bcc21058f5bc73df7088378c74372bef572e47f3"}
Sep 11 18:36:16.237896 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:16.237847    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z" podStartSLOduration=2.237832403 podStartE2EDuration="2.237832403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:16.236020444 +0000 UTC m=+1240.815285004" watchObservedRunningTime="2026-09-11 18:36:16.237832403 +0000 UTC m=+1240.817097008"
Sep 11 18:36:25.086363 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:25.086326    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:25.086835 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:25.086383    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:25.098612 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:25.098587    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:25.275772 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:25.275737    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:36:35.445470 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.445396    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"]
Sep 11 18:36:35.448801 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.448783    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:36:35.457461 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.457440    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-51815453-dockercfg-m8dkq\""
Sep 11 18:36:35.457548 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.457438    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-51815453\""
Sep 11 18:36:35.544194 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.544166    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"]
Sep 11 18:36:35.568923 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.568894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2fnx\" (UniqueName: \"kubernetes.io/projected/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-kube-api-access-l2fnx\") pod \"payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm\" (UID: \"49a2a4f6-fa76-4ea8-90dd-a062494c3ede\") " pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:36:35.569024 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.569008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-plugins-config-volume\") pod \"payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm\" (UID: \"49a2a4f6-fa76-4ea8-90dd-a062494c3ede\") " pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:36:35.670301 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.670266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-plugins-config-volume\") pod \"payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm\" (UID: \"49a2a4f6-fa76-4ea8-90dd-a062494c3ede\") " pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:36:35.670442 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.670371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fnx\" (UniqueName: \"kubernetes.io/projected/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-kube-api-access-l2fnx\") pod \"payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm\" (UID: \"49a2a4f6-fa76-4ea8-90dd-a062494c3ede\") " pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:36:35.670964 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.670946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-plugins-config-volume\") pod \"payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm\" (UID: \"49a2a4f6-fa76-4ea8-90dd-a062494c3ede\") " pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:36:35.684056 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.684031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fnx\" (UniqueName: \"kubernetes.io/projected/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-kube-api-access-l2fnx\") pod \"payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm\" (UID: \"49a2a4f6-fa76-4ea8-90dd-a062494c3ede\") " pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:36:35.758837 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:35.758772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:36:36.049459 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:36.049355    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"]
Sep 11 18:36:36.052384 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:36:36.052356    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a2a4f6_fa76_4ea8_90dd_a062494c3ede.slice/crio-0cf3618127dfcabe77db54151f376604d407a18dd88c3626a94e7cdf355f0494 WatchSource:0}: Error finding container 0cf3618127dfcabe77db54151f376604d407a18dd88c3626a94e7cdf355f0494: Status 404 returned error can't find the container with id 0cf3618127dfcabe77db54151f376604d407a18dd88c3626a94e7cdf355f0494
Sep 11 18:36:36.316861 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:36.316781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm" event={"ID":"49a2a4f6-fa76-4ea8-90dd-a062494c3ede","Type":"ContainerStarted","Data":"5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90"}
Sep 11 18:36:36.316861 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:36.316841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm" event={"ID":"49a2a4f6-fa76-4ea8-90dd-a062494c3ede","Type":"ContainerStarted","Data":"0cf3618127dfcabe77db54151f376604d407a18dd88c3626a94e7cdf355f0494"}
Sep 11 18:36:36.317033 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:36.316888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:36:36.353338 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:36.353295    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm" podStartSLOduration=1.353282567 podStartE2EDuration="1.353282567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:36.349824066 +0000 UTC m=+1260.929088626" watchObservedRunningTime="2026-09-11 18:36:36.353282567 +0000 UTC m=+1260.932547126"
Sep 11 18:36:47.324967 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:36:47.324934    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:37:15.652326 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:15.652294    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"]
Sep 11 18:37:15.652805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:15.652540    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm" podUID="49a2a4f6-fa76-4ea8-90dd-a062494c3ede" containerName="payload-processing" containerID="cri-o://5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90" gracePeriod=30
Sep 11 18:37:15.897478 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:15.897452    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:37:16.017630 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.017553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2fnx\" (UniqueName: \"kubernetes.io/projected/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-kube-api-access-l2fnx\") pod \"49a2a4f6-fa76-4ea8-90dd-a062494c3ede\" (UID: \"49a2a4f6-fa76-4ea8-90dd-a062494c3ede\") "
Sep 11 18:37:16.017794 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.017753    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-plugins-config-volume\") pod \"49a2a4f6-fa76-4ea8-90dd-a062494c3ede\" (UID: \"49a2a4f6-fa76-4ea8-90dd-a062494c3ede\") "
Sep 11 18:37:16.018184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.018139    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49a2a4f6-fa76-4ea8-90dd-a062494c3ede" (UID: "49a2a4f6-fa76-4ea8-90dd-a062494c3ede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:16.018312 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.018277    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:37:16.019605 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.019580    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-kube-api-access-l2fnx" (OuterVolumeSpecName: "kube-api-access-l2fnx") pod "49a2a4f6-fa76-4ea8-90dd-a062494c3ede" (UID: "49a2a4f6-fa76-4ea8-90dd-a062494c3ede"). InnerVolumeSpecName "kube-api-access-l2fnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:16.119330 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.119299    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2fnx\" (UniqueName: \"kubernetes.io/projected/49a2a4f6-fa76-4ea8-90dd-a062494c3ede-kube-api-access-l2fnx\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:37:16.516958 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.516922    2578 generic.go:356] "Generic (PLEG): container finished" podID="49a2a4f6-fa76-4ea8-90dd-a062494c3ede" containerID="5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90" exitCode=0
Sep 11 18:37:16.517141 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.516973    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"
Sep 11 18:37:16.517141 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.517010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm" event={"ID":"49a2a4f6-fa76-4ea8-90dd-a062494c3ede","Type":"ContainerDied","Data":"5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90"}
Sep 11 18:37:16.517141 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.517065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm" event={"ID":"49a2a4f6-fa76-4ea8-90dd-a062494c3ede","Type":"ContainerDied","Data":"0cf3618127dfcabe77db54151f376604d407a18dd88c3626a94e7cdf355f0494"}
Sep 11 18:37:16.517141 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.517091    2578 scope.go:117] "RemoveContainer" containerID="5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90"
Sep 11 18:37:16.526253 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.526236    2578 scope.go:117] "RemoveContainer" containerID="5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90"
Sep 11 18:37:16.526503 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:37:16.526483    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90\": container with ID starting with 5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90 not found: ID does not exist" containerID="5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90"
Sep 11 18:37:16.526563 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.526510    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90"} err="failed to get container status \"5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90\": rpc error: code = NotFound desc = could not find container \"5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90\": container with ID starting with 5d3da296ac7689e5b65f4ff84e7d0c76f0f426c156e6108fbf48c2646965bf90 not found: ID does not exist"
Sep 11 18:37:16.541397 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.541377    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"]
Sep 11 18:37:16.546915 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:16.546897    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51815453-b54cb7dfc-dqpfm"]
Sep 11 18:37:17.938438 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:17.938397    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49a2a4f6-fa76-4ea8-90dd-a062494c3ede" path="/var/lib/kubelet/pods/49a2a4f6-fa76-4ea8-90dd-a062494c3ede/volumes"
Sep 11 18:37:32.062485 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.062447    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"]
Sep 11 18:37:32.063137 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.063116    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49a2a4f6-fa76-4ea8-90dd-a062494c3ede" containerName="payload-processing"
Sep 11 18:37:32.063194 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.063141    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="49a2a4f6-fa76-4ea8-90dd-a062494c3ede" containerName="payload-processing"
Sep 11 18:37:32.063255 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.063243    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="49a2a4f6-fa76-4ea8-90dd-a062494c3ede" containerName="payload-processing"
Sep 11 18:37:32.067980 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.067963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.074636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.074618    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-200c07\"/\"e2e-conflict-a-ccb0c59a-kserve-self-signed-certs\""
Sep 11 18:37:32.085482 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.085454    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"]
Sep 11 18:37:32.160947 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.160915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2twxn\" (UniqueName: \"kubernetes.io/projected/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-kube-api-access-2twxn\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.160947 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.160949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-home\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.161149 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.161032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tls-certs\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.161149 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.161075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tmp-dir\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.161318 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.161170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-model-cache\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.161318 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.161211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-dshm\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.262126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.262097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tls-certs\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.262280 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.262153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tmp-dir\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.262280 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.262201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-model-cache\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.262280 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.262219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-dshm\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.262280 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.262281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2twxn\" (UniqueName: \"kubernetes.io/projected/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-kube-api-access-2twxn\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.262503 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.262301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-home\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.262596 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.262566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tmp-dir\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.262678 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.262630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-model-cache\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.262742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.262725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-home\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.264519 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.264493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-dshm\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.264719 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.264704    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tls-certs\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.271755 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.271733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2twxn\" (UniqueName: \"kubernetes.io/projected/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-kube-api-access-2twxn\") pod \"e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.376350 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.376282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:32.649791 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:32.649769    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"]
Sep 11 18:37:32.651511 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:37:32.651482    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda72fdabb_2810_4e4c_98e0_b7fb56b0613d.slice/crio-9cfe340be3e5eddf07fa32bc9c697c1fbfc071708218ef9737048617f151da24 WatchSource:0}: Error finding container 9cfe340be3e5eddf07fa32bc9c697c1fbfc071708218ef9737048617f151da24: Status 404 returned error can't find the container with id 9cfe340be3e5eddf07fa32bc9c697c1fbfc071708218ef9737048617f151da24
Sep 11 18:37:33.601547 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:33.601510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5" event={"ID":"a72fdabb-2810-4e4c-98e0-b7fb56b0613d","Type":"ContainerStarted","Data":"a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8"}
Sep 11 18:37:33.601981 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:33.601552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5" event={"ID":"a72fdabb-2810-4e4c-98e0-b7fb56b0613d","Type":"ContainerStarted","Data":"9cfe340be3e5eddf07fa32bc9c697c1fbfc071708218ef9737048617f151da24"}
Sep 11 18:37:33.632835 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:33.632788    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5" podStartSLOduration=1.632773383 podStartE2EDuration="1.632773383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:33.630271314 +0000 UTC m=+1318.209535875" watchObservedRunningTime="2026-09-11 18:37:33.632773383 +0000 UTC m=+1318.212037942"
Sep 11 18:37:42.377051 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:42.377014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:42.377430 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:42.377069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:42.389445 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:42.389422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:37:42.662752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:37:42.662669    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:38:05.752690 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.752602    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"]
Sep 11 18:38:05.756053 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.756030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:38:05.758886 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.758863    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-238bae\""
Sep 11 18:38:05.758973 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.758870    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-238bae-dockercfg-pbg4p\""
Sep 11 18:38:05.767766 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.767744    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"]
Sep 11 18:38:05.887952 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.887913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g52h\" (UniqueName: \"kubernetes.io/projected/646aa882-298f-4808-ac03-ac41a18ce9b9-kube-api-access-9g52h\") pod \"payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn\" (UID: \"646aa882-298f-4808-ac03-ac41a18ce9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:38:05.888102 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.887962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646aa882-298f-4808-ac03-ac41a18ce9b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn\" (UID: \"646aa882-298f-4808-ac03-ac41a18ce9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:38:05.989310 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.989276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g52h\" (UniqueName: \"kubernetes.io/projected/646aa882-298f-4808-ac03-ac41a18ce9b9-kube-api-access-9g52h\") pod \"payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn\" (UID: \"646aa882-298f-4808-ac03-ac41a18ce9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:38:05.989489 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.989319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646aa882-298f-4808-ac03-ac41a18ce9b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn\" (UID: \"646aa882-298f-4808-ac03-ac41a18ce9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:38:05.989950 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.989931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646aa882-298f-4808-ac03-ac41a18ce9b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn\" (UID: \"646aa882-298f-4808-ac03-ac41a18ce9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:38:05.999087 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:05.999058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g52h\" (UniqueName: \"kubernetes.io/projected/646aa882-298f-4808-ac03-ac41a18ce9b9-kube-api-access-9g52h\") pod \"payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn\" (UID: \"646aa882-298f-4808-ac03-ac41a18ce9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:38:06.066772 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:06.066745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:38:06.339828 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:06.339808    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"]
Sep 11 18:38:06.342035 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:38:06.342006    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646aa882_298f_4808_ac03_ac41a18ce9b9.slice/crio-d3033e87ddfbefbc21b2040d62c5643b50ac366a46ef905c4e19954473472874 WatchSource:0}: Error finding container d3033e87ddfbefbc21b2040d62c5643b50ac366a46ef905c4e19954473472874: Status 404 returned error can't find the container with id d3033e87ddfbefbc21b2040d62c5643b50ac366a46ef905c4e19954473472874
Sep 11 18:38:06.773769 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:06.773735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn" event={"ID":"646aa882-298f-4808-ac03-ac41a18ce9b9","Type":"ContainerStarted","Data":"24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b"}
Sep 11 18:38:06.774208 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:06.773791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn" event={"ID":"646aa882-298f-4808-ac03-ac41a18ce9b9","Type":"ContainerStarted","Data":"d3033e87ddfbefbc21b2040d62c5643b50ac366a46ef905c4e19954473472874"}
Sep 11 18:38:06.774208 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:06.773828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:38:06.794739 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:06.794694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn" podStartSLOduration=1.79468048 podStartE2EDuration="1.79468048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:06.794281583 +0000 UTC m=+1351.373546144" watchObservedRunningTime="2026-09-11 18:38:06.79468048 +0000 UTC m=+1351.373945036"
Sep 11 18:38:17.780632 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:17.780601    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:38:48.871017 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:48.870981    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"]
Sep 11 18:38:48.875173 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:48.875153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:38:48.883486 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:48.883462    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-699caf96\""
Sep 11 18:38:48.889030 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:48.889009    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"]
Sep 11 18:38:48.978058 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:48.978019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz9ln\" (UniqueName: \"kubernetes.io/projected/ac944916-4442-4528-941a-63d7f4c2fb78-kube-api-access-mz9ln\") pod \"payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8\" (UID: \"ac944916-4442-4528-941a-63d7f4c2fb78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:38:48.978058 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:48.978061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac944916-4442-4528-941a-63d7f4c2fb78-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8\" (UID: \"ac944916-4442-4528-941a-63d7f4c2fb78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:38:49.079172 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:49.079140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz9ln\" (UniqueName: \"kubernetes.io/projected/ac944916-4442-4528-941a-63d7f4c2fb78-kube-api-access-mz9ln\") pod \"payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8\" (UID: \"ac944916-4442-4528-941a-63d7f4c2fb78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:38:49.079172 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:49.079177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac944916-4442-4528-941a-63d7f4c2fb78-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8\" (UID: \"ac944916-4442-4528-941a-63d7f4c2fb78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:38:49.079904 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:49.079886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac944916-4442-4528-941a-63d7f4c2fb78-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8\" (UID: \"ac944916-4442-4528-941a-63d7f4c2fb78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:38:49.088577 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:49.088552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz9ln\" (UniqueName: \"kubernetes.io/projected/ac944916-4442-4528-941a-63d7f4c2fb78-kube-api-access-mz9ln\") pod \"payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8\" (UID: \"ac944916-4442-4528-941a-63d7f4c2fb78\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:38:49.185476 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:49.185415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:38:49.461243 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:49.461213    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"]
Sep 11 18:38:49.463036 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:38:49.463009    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac944916_4442_4528_941a_63d7f4c2fb78.slice/crio-a20d9362350c35e74303a65cb1ac32dec07cec78c4abd27ffba82be867d58f74 WatchSource:0}: Error finding container a20d9362350c35e74303a65cb1ac32dec07cec78c4abd27ffba82be867d58f74: Status 404 returned error can't find the container with id a20d9362350c35e74303a65cb1ac32dec07cec78c4abd27ffba82be867d58f74
Sep 11 18:38:49.986536 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:49.986506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8" event={"ID":"ac944916-4442-4528-941a-63d7f4c2fb78","Type":"ContainerStarted","Data":"fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9"}
Sep 11 18:38:49.986908 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:49.986553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8" event={"ID":"ac944916-4442-4528-941a-63d7f4c2fb78","Type":"ContainerStarted","Data":"a20d9362350c35e74303a65cb1ac32dec07cec78c4abd27ffba82be867d58f74"}
Sep 11 18:38:49.986908 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:49.986575    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:38:50.006692 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:38:50.006634    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8" podStartSLOduration=2.006619027 podStartE2EDuration="2.006619027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:50.004316779 +0000 UTC m=+1394.583581363" watchObservedRunningTime="2026-09-11 18:38:50.006619027 +0000 UTC m=+1394.585883589"
Sep 11 18:39:00.993521 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:00.993487    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:39:02.334939 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.334909    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"]
Sep 11 18:39:02.338553 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.338537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.341475 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.341452    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-047453\"/\"distinct-w0-047453-kserve-self-signed-certs\""
Sep 11 18:39:02.348310 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.348287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"]
Sep 11 18:39:02.503160 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.503132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tls-certs\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.503322 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.503165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slhlp\" (UniqueName: \"kubernetes.io/projected/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-kube-api-access-slhlp\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.503322 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.503249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tmp-dir\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.503322 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.503281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-model-cache\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.503322 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.503319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-home\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.503477 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.503342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-dshm\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.604535 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.604449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tls-certs\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.604535 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.604500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slhlp\" (UniqueName: \"kubernetes.io/projected/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-kube-api-access-slhlp\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.604822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.604615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tmp-dir\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.604822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.604696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-model-cache\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.604822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.604729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-home\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.604822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.604759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-dshm\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.605055 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.605034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tmp-dir\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.605132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.605109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-model-cache\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.605173 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.605148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-home\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.607000 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.606977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tls-certs\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.607101 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.607035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-dshm\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.613689 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.613638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slhlp\" (UniqueName: \"kubernetes.io/projected/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-kube-api-access-slhlp\") pod \"distinct-w0-047453-kserve-67c97f6676-msd6k\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") " pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.647949 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.647925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:02.928400 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:02.928378    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"]
Sep 11 18:39:02.930487 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:39:02.930459    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1ad4d68_5851_44e3_ab20_5228dccdfdc7.slice/crio-0e2addc299b4d0a94800ca8e8011e7bfbdef42e5f4bd9f289e33a2f1a47c77ff WatchSource:0}: Error finding container 0e2addc299b4d0a94800ca8e8011e7bfbdef42e5f4bd9f289e33a2f1a47c77ff: Status 404 returned error can't find the container with id 0e2addc299b4d0a94800ca8e8011e7bfbdef42e5f4bd9f289e33a2f1a47c77ff
Sep 11 18:39:03.054955 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:03.054919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k" event={"ID":"c1ad4d68-5851-44e3-ab20-5228dccdfdc7","Type":"ContainerStarted","Data":"92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22"}
Sep 11 18:39:03.054955 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:03.054959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k" event={"ID":"c1ad4d68-5851-44e3-ab20-5228dccdfdc7","Type":"ContainerStarted","Data":"0e2addc299b4d0a94800ca8e8011e7bfbdef42e5f4bd9f289e33a2f1a47c77ff"}
Sep 11 18:39:03.080382 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:03.080334    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k" podStartSLOduration=1.080318996 podStartE2EDuration="1.080318996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:03.078905691 +0000 UTC m=+1407.658170247" watchObservedRunningTime="2026-09-11 18:39:03.080318996 +0000 UTC m=+1407.659583557"
Sep 11 18:39:07.567168 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:07.567131    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"]
Sep 11 18:39:07.567586 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:07.567364    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c" podUID="8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb" containerName="maas-api" containerID="cri-o://b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16" gracePeriod=30
Sep 11 18:39:07.631979 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:07.631931    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"]
Sep 11 18:39:07.632198 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:07.632159    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp" podUID="b3980cc4-62a8-4e38-a426-870184f86296" containerName="payload-processing" containerID="cri-o://81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f" gracePeriod=30
Sep 11 18:39:07.639433 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:07.639404    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"]
Sep 11 18:39:07.639661 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:07.639616    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq" podUID="0deda4de-c01f-4161-bdbe-33dcafa6e04d" containerName="payload-pre-processing" containerID="cri-o://0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751" gracePeriod=30
Sep 11 18:39:07.923953 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:07.923928    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:39:07.927452 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:07.927430    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:39:07.931449 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:07.931431    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:39:08.060617 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.060587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h44g\" (UniqueName: \"kubernetes.io/projected/0deda4de-c01f-4161-bdbe-33dcafa6e04d-kube-api-access-9h44g\") pod \"0deda4de-c01f-4161-bdbe-33dcafa6e04d\" (UID: \"0deda4de-c01f-4161-bdbe-33dcafa6e04d\") "
Sep 11 18:39:08.060805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.060634    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0deda4de-c01f-4161-bdbe-33dcafa6e04d-plugins-config-volume\") pod \"0deda4de-c01f-4161-bdbe-33dcafa6e04d\" (UID: \"0deda4de-c01f-4161-bdbe-33dcafa6e04d\") "
Sep 11 18:39:08.060805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.060675    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-openshift-service-ca\") pod \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") "
Sep 11 18:39:08.060805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.060696    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-metrics-certs\") pod \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") "
Sep 11 18:39:08.060805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.060754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qwl9\" (UniqueName: \"kubernetes.io/projected/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-kube-api-access-8qwl9\") pod \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") "
Sep 11 18:39:08.061034 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.060864    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-maas-api-tls\") pod \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\" (UID: \"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb\") "
Sep 11 18:39:08.061034 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.060913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gjn5\" (UniqueName: \"kubernetes.io/projected/b3980cc4-62a8-4e38-a426-870184f86296-kube-api-access-9gjn5\") pod \"b3980cc4-62a8-4e38-a426-870184f86296\" (UID: \"b3980cc4-62a8-4e38-a426-870184f86296\") "
Sep 11 18:39:08.061034 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.060968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3980cc4-62a8-4e38-a426-870184f86296-plugins-config-volume\") pod \"b3980cc4-62a8-4e38-a426-870184f86296\" (UID: \"b3980cc4-62a8-4e38-a426-870184f86296\") "
Sep 11 18:39:08.061190 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.061097    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0deda4de-c01f-4161-bdbe-33dcafa6e04d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0deda4de-c01f-4161-bdbe-33dcafa6e04d" (UID: "0deda4de-c01f-4161-bdbe-33dcafa6e04d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:39:08.061744 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.061712    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3980cc4-62a8-4e38-a426-870184f86296-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3980cc4-62a8-4e38-a426-870184f86296" (UID: "b3980cc4-62a8-4e38-a426-870184f86296"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:39:08.061858 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.061731    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb" (UID: "8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:39:08.061921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.061874    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3980cc4-62a8-4e38-a426-870184f86296-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:39:08.061921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.061897    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0deda4de-c01f-4161-bdbe-33dcafa6e04d-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:39:08.061921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.061911    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-openshift-service-ca\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:39:08.063444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.063137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb" (UID: "8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:39:08.063444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.063300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0deda4de-c01f-4161-bdbe-33dcafa6e04d-kube-api-access-9h44g" (OuterVolumeSpecName: "kube-api-access-9h44g") pod "0deda4de-c01f-4161-bdbe-33dcafa6e04d" (UID: "0deda4de-c01f-4161-bdbe-33dcafa6e04d"). InnerVolumeSpecName "kube-api-access-9h44g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:39:08.063444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.063378    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb" (UID: "8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:39:08.063444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.063385    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-kube-api-access-8qwl9" (OuterVolumeSpecName: "kube-api-access-8qwl9") pod "8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb" (UID: "8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb"). InnerVolumeSpecName "kube-api-access-8qwl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:39:08.063757 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.063491    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3980cc4-62a8-4e38-a426-870184f86296-kube-api-access-9gjn5" (OuterVolumeSpecName: "kube-api-access-9gjn5") pod "b3980cc4-62a8-4e38-a426-870184f86296" (UID: "b3980cc4-62a8-4e38-a426-870184f86296"). InnerVolumeSpecName "kube-api-access-9gjn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:39:08.082782 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.082751    2578 generic.go:356] "Generic (PLEG): container finished" podID="8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb" containerID="b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16" exitCode=0
Sep 11 18:39:08.082956 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.082797    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"
Sep 11 18:39:08.082956 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.082830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c" event={"ID":"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb","Type":"ContainerDied","Data":"b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16"}
Sep 11 18:39:08.082956 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.082890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c" event={"ID":"8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb","Type":"ContainerDied","Data":"b71db1aa02b473e0ae724f95683eff4271d78b9b60ea6f328ae2315e78aee311"}
Sep 11 18:39:08.082956 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.082922    2578 scope.go:117] "RemoveContainer" containerID="b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16"
Sep 11 18:39:08.084211 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.084183    2578 generic.go:356] "Generic (PLEG): container finished" podID="0deda4de-c01f-4161-bdbe-33dcafa6e04d" containerID="0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751" exitCode=0
Sep 11 18:39:08.084334 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.084227    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"
Sep 11 18:39:08.084334 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.084305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq" event={"ID":"0deda4de-c01f-4161-bdbe-33dcafa6e04d","Type":"ContainerDied","Data":"0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751"}
Sep 11 18:39:08.084435 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.084347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq" event={"ID":"0deda4de-c01f-4161-bdbe-33dcafa6e04d","Type":"ContainerDied","Data":"e2389e455f10faa23d12a78a8f788549513565fc0d4652ddd5f47bbbd9595c2c"}
Sep 11 18:39:08.085857 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.085834    2578 generic.go:356] "Generic (PLEG): container finished" podID="b3980cc4-62a8-4e38-a426-870184f86296" containerID="81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f" exitCode=0
Sep 11 18:39:08.085978 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.085872    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"
Sep 11 18:39:08.085978 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.085910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp" event={"ID":"b3980cc4-62a8-4e38-a426-870184f86296","Type":"ContainerDied","Data":"81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f"}
Sep 11 18:39:08.085978 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.085959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp" event={"ID":"b3980cc4-62a8-4e38-a426-870184f86296","Type":"ContainerDied","Data":"62dfa91ff03b974daeb0861fa05bcadff6dd86302c33a51c893cef67b84df1ff"}
Sep 11 18:39:08.093753 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.093731    2578 scope.go:117] "RemoveContainer" containerID="b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16"
Sep 11 18:39:08.094048 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:39:08.094029    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16\": container with ID starting with b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16 not found: ID does not exist" containerID="b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16"
Sep 11 18:39:08.094133 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.094060    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16"} err="failed to get container status \"b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16\": rpc error: code = NotFound desc = could not find container \"b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16\": container with ID starting with b59da0c6427f5a17c875dc431061e54d5d2e583d1cdc238c2c92110ded4a2d16 not found: ID does not exist"
Sep 11 18:39:08.094133 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.094079    2578 scope.go:117] "RemoveContainer" containerID="0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751"
Sep 11 18:39:08.102052 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.102034    2578 scope.go:117] "RemoveContainer" containerID="0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751"
Sep 11 18:39:08.102294 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:39:08.102274    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751\": container with ID starting with 0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751 not found: ID does not exist" containerID="0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751"
Sep 11 18:39:08.102344 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.102302    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751"} err="failed to get container status \"0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751\": rpc error: code = NotFound desc = could not find container \"0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751\": container with ID starting with 0865facc90cd29ef7fcc94ddb545343cf3f43f8ad49f6779df170c703f531751 not found: ID does not exist"
Sep 11 18:39:08.102344 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.102318    2578 scope.go:117] "RemoveContainer" containerID="81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f"
Sep 11 18:39:08.110315 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.110299    2578 scope.go:117] "RemoveContainer" containerID="81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f"
Sep 11 18:39:08.110628 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:39:08.110602    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f\": container with ID starting with 81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f not found: ID does not exist" containerID="81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f"
Sep 11 18:39:08.110735 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.110635    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f"} err="failed to get container status \"81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f\": rpc error: code = NotFound desc = could not find container \"81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f\": container with ID starting with 81fbde6219e062394285ccbe9abfeb9d20da8f88b001d180d6bf9b8ba91e107f not found: ID does not exist"
Sep 11 18:39:08.110735 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.110722    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"]
Sep 11 18:39:08.113838 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.113816    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5a75-b7f799446-2ql5c"]
Sep 11 18:39:08.127162 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.127138    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"]
Sep 11 18:39:08.132114 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.132088    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d5a75-697bfbb486-pf9rq"]
Sep 11 18:39:08.144621 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.144595    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"]
Sep 11 18:39:08.151833 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.151813    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d5a75-664bc6dbc7-tcbnp"]
Sep 11 18:39:08.163131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.163108    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qwl9\" (UniqueName: \"kubernetes.io/projected/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-kube-api-access-8qwl9\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:39:08.163131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.163131    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-maas-api-tls\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:39:08.163286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.163141    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gjn5\" (UniqueName: \"kubernetes.io/projected/b3980cc4-62a8-4e38-a426-870184f86296-kube-api-access-9gjn5\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:39:08.163286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.163150    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h44g\" (UniqueName: \"kubernetes.io/projected/0deda4de-c01f-4161-bdbe-33dcafa6e04d-kube-api-access-9h44g\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:39:08.163286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:08.163160    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb-metrics-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:39:09.938021 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:09.937983    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0deda4de-c01f-4161-bdbe-33dcafa6e04d" path="/var/lib/kubelet/pods/0deda4de-c01f-4161-bdbe-33dcafa6e04d/volumes"
Sep 11 18:39:09.938409 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:09.938388    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb" path="/var/lib/kubelet/pods/8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb/volumes"
Sep 11 18:39:09.938768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:09.938754    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3980cc4-62a8-4e38-a426-870184f86296" path="/var/lib/kubelet/pods/b3980cc4-62a8-4e38-a426-870184f86296/volumes"
Sep 11 18:39:12.648809 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:12.648768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:12.649282 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:12.648845    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:12.666434 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:12.666403    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:13.124187 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:13.124150    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:39:34.448399 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.448323    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"]
Sep 11 18:39:34.449082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.448728    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0deda4de-c01f-4161-bdbe-33dcafa6e04d" containerName="payload-pre-processing"
Sep 11 18:39:34.449082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.448741    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0deda4de-c01f-4161-bdbe-33dcafa6e04d" containerName="payload-pre-processing"
Sep 11 18:39:34.449082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.448762    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3980cc4-62a8-4e38-a426-870184f86296" containerName="payload-processing"
Sep 11 18:39:34.449082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.448767    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3980cc4-62a8-4e38-a426-870184f86296" containerName="payload-processing"
Sep 11 18:39:34.449082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.448775    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb" containerName="maas-api"
Sep 11 18:39:34.449082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.448780    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb" containerName="maas-api"
Sep 11 18:39:34.449082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.448837    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3980cc4-62a8-4e38-a426-870184f86296" containerName="payload-processing"
Sep 11 18:39:34.449082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.448845    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d770dc7-4c9a-4e0a-aba5-92a88f2f8ceb" containerName="maas-api"
Sep 11 18:39:34.449082 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.448852    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0deda4de-c01f-4161-bdbe-33dcafa6e04d" containerName="payload-pre-processing"
Sep 11 18:39:34.451967 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.451951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.456226 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.456201    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3b7488\"/\"noten-model-3b7488-kserve-self-signed-certs\""
Sep 11 18:39:34.456405 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.456223    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3b7488\"/\"default-dockercfg-p2sgg\""
Sep 11 18:39:34.456405 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.456254    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3b7488\"/\"kube-root-ca.crt\""
Sep 11 18:39:34.456798 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.456777    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3b7488\"/\"openshift-service-ca.crt\""
Sep 11 18:39:34.470376 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.470354    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"]
Sep 11 18:39:34.504945 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.504922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzjn7\" (UniqueName: \"kubernetes.io/projected/e4da15c5-1dfe-43a2-978d-b58550214b42-kube-api-access-fzjn7\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.505069 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.505009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-dshm\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.505069 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.505031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-model-cache\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.505069 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.505063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-tmp-dir\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.505194 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.505114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4da15c5-1dfe-43a2-978d-b58550214b42-tls-certs\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.505194 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.505148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-home\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.606430 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.606402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-dshm\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.606535 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.606435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-model-cache\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.606535 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.606461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-tmp-dir\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.606613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.606570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4da15c5-1dfe-43a2-978d-b58550214b42-tls-certs\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.606613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.606600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-home\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.606740 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.606679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzjn7\" (UniqueName: \"kubernetes.io/projected/e4da15c5-1dfe-43a2-978d-b58550214b42-kube-api-access-fzjn7\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.606799 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.606769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-tmp-dir\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.606891 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.606873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-model-cache\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.607012 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.606983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-home\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.608602 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.608577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-dshm\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.608925 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.608910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4da15c5-1dfe-43a2-978d-b58550214b42-tls-certs\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.616489 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.616467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzjn7\" (UniqueName: \"kubernetes.io/projected/e4da15c5-1dfe-43a2-978d-b58550214b42-kube-api-access-fzjn7\") pod \"noten-model-3b7488-kserve-9b88dcf95-hpqh2\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") " pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:34.761627 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:34.761551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:35.036532 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:35.036501    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"]
Sep 11 18:39:35.039839 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:39:35.039812    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4da15c5_1dfe_43a2_978d_b58550214b42.slice/crio-8937df7a5310d2eb48b6b8a3a756bff07a5082bfeb4a6f7e6e9f2348e74d3ac6 WatchSource:0}: Error finding container 8937df7a5310d2eb48b6b8a3a756bff07a5082bfeb4a6f7e6e9f2348e74d3ac6: Status 404 returned error can't find the container with id 8937df7a5310d2eb48b6b8a3a756bff07a5082bfeb4a6f7e6e9f2348e74d3ac6
Sep 11 18:39:35.220932 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:35.220897    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2" event={"ID":"e4da15c5-1dfe-43a2-978d-b58550214b42","Type":"ContainerStarted","Data":"6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d"}
Sep 11 18:39:35.221101 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:35.220940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2" event={"ID":"e4da15c5-1dfe-43a2-978d-b58550214b42","Type":"ContainerStarted","Data":"8937df7a5310d2eb48b6b8a3a756bff07a5082bfeb4a6f7e6e9f2348e74d3ac6"}
Sep 11 18:39:35.243378 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:35.243332    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2" podStartSLOduration=1.243317756 podStartE2EDuration="1.243317756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:35.241030558 +0000 UTC m=+1439.820295118" watchObservedRunningTime="2026-09-11 18:39:35.243317756 +0000 UTC m=+1439.822582364"
Sep 11 18:39:44.761685 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:44.761636    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:44.762158 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:44.761704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:44.774027 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:44.774001    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:45.281076 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:45.281044    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:39:54.166613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.166579    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"]
Sep 11 18:39:54.171484 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.171467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.175909 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.175886    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-200c07\"/\"e2e-conflict-b-ccb0c59a-kserve-self-signed-certs\""
Sep 11 18:39:54.185798 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.185765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"]
Sep 11 18:39:54.210046 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.209990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-tmp-dir\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.210198 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.210047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-model-cache\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.210198 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.210186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc41905c-ac79-4776-99d6-20ee7934b102-tls-certs\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.210340 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.210254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-home\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.210340 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.210294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-dshm\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.210448 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.210360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8lz\" (UniqueName: \"kubernetes.io/projected/fc41905c-ac79-4776-99d6-20ee7934b102-kube-api-access-wd8lz\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.311754 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.311730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc41905c-ac79-4776-99d6-20ee7934b102-tls-certs\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.311897 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.311789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-home\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.311897 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.311828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-dshm\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.311897 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.311876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8lz\" (UniqueName: \"kubernetes.io/projected/fc41905c-ac79-4776-99d6-20ee7934b102-kube-api-access-wd8lz\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.312057 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.311927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-tmp-dir\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.312057 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.311978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-model-cache\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.312319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.312287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-home\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.312319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.312315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-tmp-dir\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.312492 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.312363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-model-cache\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.314074 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.314045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-dshm\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.314341 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.314321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc41905c-ac79-4776-99d6-20ee7934b102-tls-certs\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.320199 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.320177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8lz\" (UniqueName: \"kubernetes.io/projected/fc41905c-ac79-4776-99d6-20ee7934b102-kube-api-access-wd8lz\") pod \"e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") " pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.481282 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.481213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:39:54.754835 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:54.754762    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"]
Sep 11 18:39:54.758046 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:39:54.758014    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc41905c_ac79_4776_99d6_20ee7934b102.slice/crio-02892a1431c415d797e3e29877256bdfcd1ac6f1e13f6318415cbfa9a20136d7 WatchSource:0}: Error finding container 02892a1431c415d797e3e29877256bdfcd1ac6f1e13f6318415cbfa9a20136d7: Status 404 returned error can't find the container with id 02892a1431c415d797e3e29877256bdfcd1ac6f1e13f6318415cbfa9a20136d7
Sep 11 18:39:55.320583 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:55.320550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87" event={"ID":"fc41905c-ac79-4776-99d6-20ee7934b102","Type":"ContainerStarted","Data":"829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443"}
Sep 11 18:39:55.320940 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:55.320591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87" event={"ID":"fc41905c-ac79-4776-99d6-20ee7934b102","Type":"ContainerStarted","Data":"02892a1431c415d797e3e29877256bdfcd1ac6f1e13f6318415cbfa9a20136d7"}
Sep 11 18:39:55.355066 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:55.355001    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87" podStartSLOduration=1.354981273 podStartE2EDuration="1.354981273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:55.352202685 +0000 UTC m=+1459.931467259" watchObservedRunningTime="2026-09-11 18:39:55.354981273 +0000 UTC m=+1459.934245835"
Sep 11 18:39:56.945441 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:39:56.945408    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"]
Sep 11 18:40:01.598026 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:01.597994    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"]
Sep 11 18:40:01.598482 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:01.598235    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4" podUID="cf238a2d-f761-4475-9397-51f024a57658" containerName="payload-processing" containerID="cri-o://131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a" gracePeriod=30
Sep 11 18:40:01.849368 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:01.849303    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:40:01.981967 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:01.981937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nlbd\" (UniqueName: \"kubernetes.io/projected/cf238a2d-f761-4475-9397-51f024a57658-kube-api-access-5nlbd\") pod \"cf238a2d-f761-4475-9397-51f024a57658\" (UID: \"cf238a2d-f761-4475-9397-51f024a57658\") "
Sep 11 18:40:01.982126 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:01.982041    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf238a2d-f761-4475-9397-51f024a57658-plugins-config-volume\") pod \"cf238a2d-f761-4475-9397-51f024a57658\" (UID: \"cf238a2d-f761-4475-9397-51f024a57658\") "
Sep 11 18:40:01.982451 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:01.982418    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf238a2d-f761-4475-9397-51f024a57658-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf238a2d-f761-4475-9397-51f024a57658" (UID: "cf238a2d-f761-4475-9397-51f024a57658"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:40:01.984071 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:01.984047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf238a2d-f761-4475-9397-51f024a57658-kube-api-access-5nlbd" (OuterVolumeSpecName: "kube-api-access-5nlbd") pod "cf238a2d-f761-4475-9397-51f024a57658" (UID: "cf238a2d-f761-4475-9397-51f024a57658"). InnerVolumeSpecName "kube-api-access-5nlbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:40:02.083769 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.083731    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf238a2d-f761-4475-9397-51f024a57658-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:02.083769 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.083763    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nlbd\" (UniqueName: \"kubernetes.io/projected/cf238a2d-f761-4475-9397-51f024a57658-kube-api-access-5nlbd\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:02.354799 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.354770    2578 generic.go:356] "Generic (PLEG): container finished" podID="cf238a2d-f761-4475-9397-51f024a57658" containerID="131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a" exitCode=0
Sep 11 18:40:02.354983 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.354821    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"
Sep 11 18:40:02.354983 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.354838    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4" event={"ID":"cf238a2d-f761-4475-9397-51f024a57658","Type":"ContainerDied","Data":"131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a"}
Sep 11 18:40:02.354983 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.354879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4" event={"ID":"cf238a2d-f761-4475-9397-51f024a57658","Type":"ContainerDied","Data":"742fae16f1f2ba6f0fa71afa9ad4305b7034f7dd375012d616d7720206a12667"}
Sep 11 18:40:02.354983 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.354899    2578 scope.go:117] "RemoveContainer" containerID="131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a"
Sep 11 18:40:02.363879 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.363861    2578 scope.go:117] "RemoveContainer" containerID="131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a"
Sep 11 18:40:02.364140 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:40:02.364113    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a\": container with ID starting with 131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a not found: ID does not exist" containerID="131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a"
Sep 11 18:40:02.364212 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.364147    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a"} err="failed to get container status \"131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a\": rpc error: code = NotFound desc = could not find container \"131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a\": container with ID starting with 131a22933a204e20ba94b9b151bd5d7c7f6dac6bc8a9ae5299ee7f1aade23d2a not found: ID does not exist"
Sep 11 18:40:02.379788 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.379763    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"]
Sep 11 18:40:02.385658 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:02.385618    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e8f201-79866b68c5-fq9v4"]
Sep 11 18:40:03.937931 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:03.937896    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf238a2d-f761-4475-9397-51f024a57658" path="/var/lib/kubelet/pods/cf238a2d-f761-4475-9397-51f024a57658/volumes"
Sep 11 18:40:04.481791 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:04.481746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:40:04.481791 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:04.481806    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:40:04.494662 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:04.494622    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:40:05.381604 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:05.381573    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:40:11.963808 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:11.963757    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z" podUID="31f20dfd-06b2-4d49-8d46-38474daa3d3a" containerName="main" containerID="cri-o://fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909" gracePeriod=45
Sep 11 18:40:12.217836 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.217782    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:40:12.281607 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.281572    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tmp-dir\") pod \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") "
Sep 11 18:40:12.281805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.281694    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-dshm\") pod \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") "
Sep 11 18:40:12.281805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.281726    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tls-certs\") pod \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") "
Sep 11 18:40:12.281805 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.281768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-home\") pod \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") "
Sep 11 18:40:12.281966 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.281842    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xj7t\" (UniqueName: \"kubernetes.io/projected/31f20dfd-06b2-4d49-8d46-38474daa3d3a-kube-api-access-7xj7t\") pod \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") "
Sep 11 18:40:12.281966 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.281899    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-model-cache\") pod \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\" (UID: \"31f20dfd-06b2-4d49-8d46-38474daa3d3a\") "
Sep 11 18:40:12.282062 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.281892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31f20dfd-06b2-4d49-8d46-38474daa3d3a" (UID: "31f20dfd-06b2-4d49-8d46-38474daa3d3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:40:12.282113 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.282055    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-home" (OuterVolumeSpecName: "home") pod "31f20dfd-06b2-4d49-8d46-38474daa3d3a" (UID: "31f20dfd-06b2-4d49-8d46-38474daa3d3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:40:12.282170 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.282148    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-model-cache" (OuterVolumeSpecName: "model-cache") pod "31f20dfd-06b2-4d49-8d46-38474daa3d3a" (UID: "31f20dfd-06b2-4d49-8d46-38474daa3d3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:40:12.282300 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.282282    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:12.282363 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.282305    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:12.282363 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.282320    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:12.283920 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.283897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-dshm" (OuterVolumeSpecName: "dshm") pod "31f20dfd-06b2-4d49-8d46-38474daa3d3a" (UID: "31f20dfd-06b2-4d49-8d46-38474daa3d3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:40:12.284012 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.283949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31f20dfd-06b2-4d49-8d46-38474daa3d3a" (UID: "31f20dfd-06b2-4d49-8d46-38474daa3d3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:40:12.284012 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.283960    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f20dfd-06b2-4d49-8d46-38474daa3d3a-kube-api-access-7xj7t" (OuterVolumeSpecName: "kube-api-access-7xj7t") pod "31f20dfd-06b2-4d49-8d46-38474daa3d3a" (UID: "31f20dfd-06b2-4d49-8d46-38474daa3d3a"). InnerVolumeSpecName "kube-api-access-7xj7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:40:12.383132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.383107    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f20dfd-06b2-4d49-8d46-38474daa3d3a-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:12.383132 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.383130    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f20dfd-06b2-4d49-8d46-38474daa3d3a-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:12.383292 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.383140    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xj7t\" (UniqueName: \"kubernetes.io/projected/31f20dfd-06b2-4d49-8d46-38474daa3d3a-kube-api-access-7xj7t\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:12.406738 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.406706    2578 generic.go:356] "Generic (PLEG): container finished" podID="31f20dfd-06b2-4d49-8d46-38474daa3d3a" containerID="fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909" exitCode=0
Sep 11 18:40:12.406848 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.406748    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"
Sep 11 18:40:12.406848 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.406761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z" event={"ID":"31f20dfd-06b2-4d49-8d46-38474daa3d3a","Type":"ContainerDied","Data":"fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909"}
Sep 11 18:40:12.406848 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.406792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z" event={"ID":"31f20dfd-06b2-4d49-8d46-38474daa3d3a","Type":"ContainerDied","Data":"9b587c0c09c4d19e65574783bcc21058f5bc73df7088378c74372bef572e47f3"}
Sep 11 18:40:12.406848 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.406812    2578 scope.go:117] "RemoveContainer" containerID="fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909"
Sep 11 18:40:12.416147 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.416128    2578 scope.go:117] "RemoveContainer" containerID="fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909"
Sep 11 18:40:12.416428 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:40:12.416409    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909\": container with ID starting with fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909 not found: ID does not exist" containerID="fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909"
Sep 11 18:40:12.416504 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.416434    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909"} err="failed to get container status \"fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909\": rpc error: code = NotFound desc = could not find container \"fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909\": container with ID starting with fdf31a5e3f8c03e584bbe32f2f05ae3505ff2e20279c11b527de1b089623a909 not found: ID does not exist"
Sep 11 18:40:12.430171 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.430147    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"]
Sep 11 18:40:12.435069 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.435049    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e8f201/test-model-e8f201-kserve-5c586bb95c-nvk8z"]
Sep 11 18:40:12.898524 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:12.898493    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"]
Sep 11 18:40:13.938147 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:13.938063    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31f20dfd-06b2-4d49-8d46-38474daa3d3a" path="/var/lib/kubelet/pods/31f20dfd-06b2-4d49-8d46-38474daa3d3a/volumes"
Sep 11 18:40:27.917434 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:27.917364    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2" podUID="e4da15c5-1dfe-43a2-978d-b58550214b42" containerName="main" containerID="cri-o://6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d" gracePeriod=45
Sep 11 18:40:28.181537 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.181480    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:40:28.229203 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.229173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-tmp-dir\") pod \"e4da15c5-1dfe-43a2-978d-b58550214b42\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") "
Sep 11 18:40:28.229389 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.229243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzjn7\" (UniqueName: \"kubernetes.io/projected/e4da15c5-1dfe-43a2-978d-b58550214b42-kube-api-access-fzjn7\") pod \"e4da15c5-1dfe-43a2-978d-b58550214b42\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") "
Sep 11 18:40:28.229389 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.229287    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-home\") pod \"e4da15c5-1dfe-43a2-978d-b58550214b42\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") "
Sep 11 18:40:28.229389 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.229325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-model-cache\") pod \"e4da15c5-1dfe-43a2-978d-b58550214b42\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") "
Sep 11 18:40:28.229581 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.229465    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4da15c5-1dfe-43a2-978d-b58550214b42" (UID: "e4da15c5-1dfe-43a2-978d-b58550214b42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:40:28.229581 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.229495    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4da15c5-1dfe-43a2-978d-b58550214b42" (UID: "e4da15c5-1dfe-43a2-978d-b58550214b42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:40:28.229729 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.229619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-dshm\") pod \"e4da15c5-1dfe-43a2-978d-b58550214b42\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") "
Sep 11 18:40:28.229729 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.229644    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-home" (OuterVolumeSpecName: "home") pod "e4da15c5-1dfe-43a2-978d-b58550214b42" (UID: "e4da15c5-1dfe-43a2-978d-b58550214b42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:40:28.229851 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.229766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4da15c5-1dfe-43a2-978d-b58550214b42-tls-certs\") pod \"e4da15c5-1dfe-43a2-978d-b58550214b42\" (UID: \"e4da15c5-1dfe-43a2-978d-b58550214b42\") "
Sep 11 18:40:28.230169 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.230147    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:28.230272 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.230174    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:28.230272 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.230212    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:28.235076 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.235029    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-dshm" (OuterVolumeSpecName: "dshm") pod "e4da15c5-1dfe-43a2-978d-b58550214b42" (UID: "e4da15c5-1dfe-43a2-978d-b58550214b42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:40:28.235076 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.235053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4da15c5-1dfe-43a2-978d-b58550214b42-kube-api-access-fzjn7" (OuterVolumeSpecName: "kube-api-access-fzjn7") pod "e4da15c5-1dfe-43a2-978d-b58550214b42" (UID: "e4da15c5-1dfe-43a2-978d-b58550214b42"). InnerVolumeSpecName "kube-api-access-fzjn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:40:28.235076 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.235032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4da15c5-1dfe-43a2-978d-b58550214b42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4da15c5-1dfe-43a2-978d-b58550214b42" (UID: "e4da15c5-1dfe-43a2-978d-b58550214b42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:40:28.330985 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.330952    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzjn7\" (UniqueName: \"kubernetes.io/projected/e4da15c5-1dfe-43a2-978d-b58550214b42-kube-api-access-fzjn7\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:28.330985 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.330978    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4da15c5-1dfe-43a2-978d-b58550214b42-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:28.330985 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.330987    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4da15c5-1dfe-43a2-978d-b58550214b42-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:40:28.484753 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.484642    2578 generic.go:356] "Generic (PLEG): container finished" podID="e4da15c5-1dfe-43a2-978d-b58550214b42" containerID="6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d" exitCode=0
Sep 11 18:40:28.484753 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.484694    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"
Sep 11 18:40:28.484753 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.484741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2" event={"ID":"e4da15c5-1dfe-43a2-978d-b58550214b42","Type":"ContainerDied","Data":"6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d"}
Sep 11 18:40:28.485020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.484791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2" event={"ID":"e4da15c5-1dfe-43a2-978d-b58550214b42","Type":"ContainerDied","Data":"8937df7a5310d2eb48b6b8a3a756bff07a5082bfeb4a6f7e6e9f2348e74d3ac6"}
Sep 11 18:40:28.485020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.484813    2578 scope.go:117] "RemoveContainer" containerID="6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d"
Sep 11 18:40:28.493633 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.493614    2578 scope.go:117] "RemoveContainer" containerID="6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d"
Sep 11 18:40:28.493913 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:40:28.493895    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d\": container with ID starting with 6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d not found: ID does not exist" containerID="6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d"
Sep 11 18:40:28.493963 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.493923    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d"} err="failed to get container status \"6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d\": rpc error: code = NotFound desc = could not find container \"6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d\": container with ID starting with 6b150ec3a8753daf40087d372e248508d86d31a6c1e7344f82d832837735a97d not found: ID does not exist"
Sep 11 18:40:28.507396 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.507361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"]
Sep 11 18:40:28.512612 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:28.512587    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-3b7488/noten-model-3b7488-kserve-9b88dcf95-hpqh2"]
Sep 11 18:40:29.938260 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:29.938226    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4da15c5-1dfe-43a2-978d-b58550214b42" path="/var/lib/kubelet/pods/e4da15c5-1dfe-43a2-978d-b58550214b42/volumes"
Sep 11 18:40:52.563048 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:52.563013    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"]
Sep 11 18:40:52.568430 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:52.568398    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"]
Sep 11 18:40:53.910415 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.910375    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"]
Sep 11 18:40:53.910798 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.910783    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4da15c5-1dfe-43a2-978d-b58550214b42" containerName="main"
Sep 11 18:40:53.910798 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.910796    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4da15c5-1dfe-43a2-978d-b58550214b42" containerName="main"
Sep 11 18:40:53.910875 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.910812    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf238a2d-f761-4475-9397-51f024a57658" containerName="payload-processing"
Sep 11 18:40:53.910875 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.910818    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf238a2d-f761-4475-9397-51f024a57658" containerName="payload-processing"
Sep 11 18:40:53.910875 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.910828    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f20dfd-06b2-4d49-8d46-38474daa3d3a" containerName="main"
Sep 11 18:40:53.910875 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.910833    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f20dfd-06b2-4d49-8d46-38474daa3d3a" containerName="main"
Sep 11 18:40:53.910994 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.910892    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4da15c5-1dfe-43a2-978d-b58550214b42" containerName="main"
Sep 11 18:40:53.910994 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.910900    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f20dfd-06b2-4d49-8d46-38474daa3d3a" containerName="main"
Sep 11 18:40:53.910994 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.910910    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf238a2d-f761-4475-9397-51f024a57658" containerName="payload-processing"
Sep 11 18:40:53.914057 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.914042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:40:53.916896 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.916876    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d90e71\""
Sep 11 18:40:53.924698 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.924676    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"]
Sep 11 18:40:53.947487 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.947464    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"]
Sep 11 18:40:53.951292 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.951274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:40:53.954072 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.954054    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d90e71-dockercfg-fnm4v\""
Sep 11 18:40:53.964399 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:53.964381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"]
Sep 11 18:40:54.081578 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.081543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsqn8\" (UniqueName: \"kubernetes.io/projected/a544c074-ed04-40a5-b41b-93f8cfbbfb66-kube-api-access-hsqn8\") pod \"payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4\" (UID: \"a544c074-ed04-40a5-b41b-93f8cfbbfb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:40:54.081777 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.081602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qr5h\" (UniqueName: \"kubernetes.io/projected/636e0fa1-283e-4e81-96b8-070852063a4b-kube-api-access-5qr5h\") pod \"payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6\" (UID: \"636e0fa1-283e-4e81-96b8-070852063a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:40:54.081777 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.081729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636e0fa1-283e-4e81-96b8-070852063a4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6\" (UID: \"636e0fa1-283e-4e81-96b8-070852063a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:40:54.081915 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.081894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a544c074-ed04-40a5-b41b-93f8cfbbfb66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4\" (UID: \"a544c074-ed04-40a5-b41b-93f8cfbbfb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:40:54.183003 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.182925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsqn8\" (UniqueName: \"kubernetes.io/projected/a544c074-ed04-40a5-b41b-93f8cfbbfb66-kube-api-access-hsqn8\") pod \"payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4\" (UID: \"a544c074-ed04-40a5-b41b-93f8cfbbfb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:40:54.183003 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.182982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qr5h\" (UniqueName: \"kubernetes.io/projected/636e0fa1-283e-4e81-96b8-070852063a4b-kube-api-access-5qr5h\") pod \"payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6\" (UID: \"636e0fa1-283e-4e81-96b8-070852063a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:40:54.183214 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.183015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636e0fa1-283e-4e81-96b8-070852063a4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6\" (UID: \"636e0fa1-283e-4e81-96b8-070852063a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:40:54.183214 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.183061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a544c074-ed04-40a5-b41b-93f8cfbbfb66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4\" (UID: \"a544c074-ed04-40a5-b41b-93f8cfbbfb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:40:54.183641 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.183621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636e0fa1-283e-4e81-96b8-070852063a4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6\" (UID: \"636e0fa1-283e-4e81-96b8-070852063a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:40:54.183735 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.183716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a544c074-ed04-40a5-b41b-93f8cfbbfb66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4\" (UID: \"a544c074-ed04-40a5-b41b-93f8cfbbfb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:40:54.194262 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.194228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsqn8\" (UniqueName: \"kubernetes.io/projected/a544c074-ed04-40a5-b41b-93f8cfbbfb66-kube-api-access-hsqn8\") pod \"payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4\" (UID: \"a544c074-ed04-40a5-b41b-93f8cfbbfb66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:40:54.195041 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.195016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qr5h\" (UniqueName: \"kubernetes.io/projected/636e0fa1-283e-4e81-96b8-070852063a4b-kube-api-access-5qr5h\") pod \"payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6\" (UID: \"636e0fa1-283e-4e81-96b8-070852063a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:40:54.224857 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.224833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:40:54.261085 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.261057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:40:54.503919 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.503898    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"]
Sep 11 18:40:54.505684 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:40:54.505634    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda544c074_ed04_40a5_b41b_93f8cfbbfb66.slice/crio-c3c60514a3cb9ad175c72d7bd5a0b80409350632d9004d224ffb80e7da1e01b6 WatchSource:0}: Error finding container c3c60514a3cb9ad175c72d7bd5a0b80409350632d9004d224ffb80e7da1e01b6: Status 404 returned error can't find the container with id c3c60514a3cb9ad175c72d7bd5a0b80409350632d9004d224ffb80e7da1e01b6
Sep 11 18:40:54.551762 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.551736    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"]
Sep 11 18:40:54.552450 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:40:54.552424    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636e0fa1_283e_4e81_96b8_070852063a4b.slice/crio-ac7bf1eac0624ca52e223bd9050b2bd27edb94c0dc2675812297183031a96b59 WatchSource:0}: Error finding container ac7bf1eac0624ca52e223bd9050b2bd27edb94c0dc2675812297183031a96b59: Status 404 returned error can't find the container with id ac7bf1eac0624ca52e223bd9050b2bd27edb94c0dc2675812297183031a96b59
Sep 11 18:40:54.611808 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.611781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6" event={"ID":"636e0fa1-283e-4e81-96b8-070852063a4b","Type":"ContainerStarted","Data":"ac7bf1eac0624ca52e223bd9050b2bd27edb94c0dc2675812297183031a96b59"}
Sep 11 18:40:54.613438 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.613412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4" event={"ID":"a544c074-ed04-40a5-b41b-93f8cfbbfb66","Type":"ContainerStarted","Data":"7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974"}
Sep 11 18:40:54.613558 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.613454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4" event={"ID":"a544c074-ed04-40a5-b41b-93f8cfbbfb66","Type":"ContainerStarted","Data":"c3c60514a3cb9ad175c72d7bd5a0b80409350632d9004d224ffb80e7da1e01b6"}
Sep 11 18:40:54.613558 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.613510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:40:54.631238 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:54.631185    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4" podStartSLOduration=1.631165444 podStartE2EDuration="1.631165444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:40:54.630795026 +0000 UTC m=+1519.210059588" watchObservedRunningTime="2026-09-11 18:40:54.631165444 +0000 UTC m=+1519.210430005"
Sep 11 18:40:55.619886 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:55.619850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6" event={"ID":"636e0fa1-283e-4e81-96b8-070852063a4b","Type":"ContainerStarted","Data":"c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d"}
Sep 11 18:40:55.620268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:55.619926    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:40:55.645463 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:40:55.645402    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6" podStartSLOduration=2.645381441 podStartE2EDuration="2.645381441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:40:55.64201932 +0000 UTC m=+1520.221283884" watchObservedRunningTime="2026-09-11 18:40:55.645381441 +0000 UTC m=+1520.224646003"
Sep 11 18:41:03.737267 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:03.737185    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"]
Sep 11 18:41:03.737680 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:03.737436    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq" podUID="1053b8ae-1501-46f8-8b2f-b0a3488bd261" containerName="maas-api" containerID="cri-o://d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1" gracePeriod=30
Sep 11 18:41:03.992993 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:03.992936    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:41:04.071584 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.071551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-maas-api-tls\") pod \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") "
Sep 11 18:41:04.071797 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.071758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-metrics-certs\") pod \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") "
Sep 11 18:41:04.071875 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.071823    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmwb7\" (UniqueName: \"kubernetes.io/projected/1053b8ae-1501-46f8-8b2f-b0a3488bd261-kube-api-access-vmwb7\") pod \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") "
Sep 11 18:41:04.071875 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.071849    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1053b8ae-1501-46f8-8b2f-b0a3488bd261-openshift-service-ca\") pod \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\" (UID: \"1053b8ae-1501-46f8-8b2f-b0a3488bd261\") "
Sep 11 18:41:04.072315 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.072276    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1053b8ae-1501-46f8-8b2f-b0a3488bd261-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1053b8ae-1501-46f8-8b2f-b0a3488bd261" (UID: "1053b8ae-1501-46f8-8b2f-b0a3488bd261"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:04.073809 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.073780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1053b8ae-1501-46f8-8b2f-b0a3488bd261" (UID: "1053b8ae-1501-46f8-8b2f-b0a3488bd261"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:04.073928 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.073803    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1053b8ae-1501-46f8-8b2f-b0a3488bd261" (UID: "1053b8ae-1501-46f8-8b2f-b0a3488bd261"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:04.073928 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.073872    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1053b8ae-1501-46f8-8b2f-b0a3488bd261-kube-api-access-vmwb7" (OuterVolumeSpecName: "kube-api-access-vmwb7") pod "1053b8ae-1501-46f8-8b2f-b0a3488bd261" (UID: "1053b8ae-1501-46f8-8b2f-b0a3488bd261"). InnerVolumeSpecName "kube-api-access-vmwb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:04.173286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.173259    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-maas-api-tls\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:04.173286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.173282    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1053b8ae-1501-46f8-8b2f-b0a3488bd261-metrics-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:04.173286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.173299    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmwb7\" (UniqueName: \"kubernetes.io/projected/1053b8ae-1501-46f8-8b2f-b0a3488bd261-kube-api-access-vmwb7\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:04.173504 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.173310    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1053b8ae-1501-46f8-8b2f-b0a3488bd261-openshift-service-ca\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:04.665328 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.665292    2578 generic.go:356] "Generic (PLEG): container finished" podID="1053b8ae-1501-46f8-8b2f-b0a3488bd261" containerID="d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1" exitCode=0
Sep 11 18:41:04.665495 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.665339    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"
Sep 11 18:41:04.665495 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.665372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq" event={"ID":"1053b8ae-1501-46f8-8b2f-b0a3488bd261","Type":"ContainerDied","Data":"d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1"}
Sep 11 18:41:04.665495 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.665421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq" event={"ID":"1053b8ae-1501-46f8-8b2f-b0a3488bd261","Type":"ContainerDied","Data":"b7f9e7c1c7e8285807b5c0236c23d39369d2a01ed9a2ceee5abcd6df35d11b47"}
Sep 11 18:41:04.665495 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.665442    2578 scope.go:117] "RemoveContainer" containerID="d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1"
Sep 11 18:41:04.675222 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.675206    2578 scope.go:117] "RemoveContainer" containerID="d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1"
Sep 11 18:41:04.675489 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:41:04.675469    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1\": container with ID starting with d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1 not found: ID does not exist" containerID="d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1"
Sep 11 18:41:04.675537 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.675500    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1"} err="failed to get container status \"d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1\": rpc error: code = NotFound desc = could not find container \"d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1\": container with ID starting with d10c1da92c0a704e0dcb3e45b5f0cc91cb590b5ef636392ee7d8215d8768bba1 not found: ID does not exist"
Sep 11 18:41:04.694713 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.694691    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"]
Sep 11 18:41:04.719862 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:04.719836    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-047453-5949858f74-qb8bq"]
Sep 11 18:41:05.622041 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:05.622009    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:41:05.939077 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:05.938999    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1053b8ae-1501-46f8-8b2f-b0a3488bd261" path="/var/lib/kubelet/pods/1053b8ae-1501-46f8-8b2f-b0a3488bd261/volumes"
Sep 11 18:41:06.627200 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:06.627161    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:41:07.581926 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.581889    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k" podUID="c1ad4d68-5851-44e3-ab20-5228dccdfdc7" containerName="main" containerID="cri-o://92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22" gracePeriod=45
Sep 11 18:41:07.586613 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.586585    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z" podUID="9dfe1397-45ce-454d-b9d1-af3c7fce823a" containerName="main" containerID="cri-o://50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1" gracePeriod=45
Sep 11 18:41:07.851951 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.851927    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:41:07.854934 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.854914    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:41:07.908218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908184    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-model-cache\") pod \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") "
Sep 11 18:41:07.908407 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908292    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-dshm\") pod \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") "
Sep 11 18:41:07.908407 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908330    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-home\") pod \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") "
Sep 11 18:41:07.908407 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slhlp\" (UniqueName: \"kubernetes.io/projected/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-kube-api-access-slhlp\") pod \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") "
Sep 11 18:41:07.908592 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tls-certs\") pod \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") "
Sep 11 18:41:07.908592 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908454    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1ad4d68-5851-44e3-ab20-5228dccdfdc7" (UID: "c1ad4d68-5851-44e3-ab20-5228dccdfdc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:07.908592 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tmp-dir\") pod \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") "
Sep 11 18:41:07.908592 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908534    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tmp-dir\") pod \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") "
Sep 11 18:41:07.908833 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9dfe1397-45ce-454d-b9d1-af3c7fce823a" (UID: "9dfe1397-45ce-454d-b9d1-af3c7fce823a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:07.908833 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908789    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-home\") pod \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") "
Sep 11 18:41:07.908945 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908841    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1ad4d68-5851-44e3-ab20-5228dccdfdc7" (UID: "c1ad4d68-5851-44e3-ab20-5228dccdfdc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:07.908945 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908863    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5g7h\" (UniqueName: \"kubernetes.io/projected/9dfe1397-45ce-454d-b9d1-af3c7fce823a-kube-api-access-c5g7h\") pod \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") "
Sep 11 18:41:07.908945 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908896    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-dshm\") pod \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") "
Sep 11 18:41:07.908945 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908921    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-model-cache\") pod \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\" (UID: \"9dfe1397-45ce-454d-b9d1-af3c7fce823a\") "
Sep 11 18:41:07.909150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908954    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tls-certs\") pod \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\" (UID: \"c1ad4d68-5851-44e3-ab20-5228dccdfdc7\") "
Sep 11 18:41:07.909150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.908995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-home" (OuterVolumeSpecName: "home") pod "9dfe1397-45ce-454d-b9d1-af3c7fce823a" (UID: "9dfe1397-45ce-454d-b9d1-af3c7fce823a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:07.909150 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.909040    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-home" (OuterVolumeSpecName: "home") pod "c1ad4d68-5851-44e3-ab20-5228dccdfdc7" (UID: "c1ad4d68-5851-44e3-ab20-5228dccdfdc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:07.909305 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.909207    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9dfe1397-45ce-454d-b9d1-af3c7fce823a" (UID: "9dfe1397-45ce-454d-b9d1-af3c7fce823a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:07.909362 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.909334    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:07.909362 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.909353    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:07.909470 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.909366    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:07.909470 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.909378    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:07.909470 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.909391    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:07.909470 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.909402    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:07.911611 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.911575    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-dshm" (OuterVolumeSpecName: "dshm") pod "9dfe1397-45ce-454d-b9d1-af3c7fce823a" (UID: "9dfe1397-45ce-454d-b9d1-af3c7fce823a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:07.911946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.911867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9dfe1397-45ce-454d-b9d1-af3c7fce823a" (UID: "9dfe1397-45ce-454d-b9d1-af3c7fce823a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:07.911946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.911914    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1ad4d68-5851-44e3-ab20-5228dccdfdc7" (UID: "c1ad4d68-5851-44e3-ab20-5228dccdfdc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:07.912202 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.912184    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-kube-api-access-slhlp" (OuterVolumeSpecName: "kube-api-access-slhlp") pod "c1ad4d68-5851-44e3-ab20-5228dccdfdc7" (UID: "c1ad4d68-5851-44e3-ab20-5228dccdfdc7"). InnerVolumeSpecName "kube-api-access-slhlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:07.912483 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.912461    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-dshm" (OuterVolumeSpecName: "dshm") pod "c1ad4d68-5851-44e3-ab20-5228dccdfdc7" (UID: "c1ad4d68-5851-44e3-ab20-5228dccdfdc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:07.912483 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:07.912469    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfe1397-45ce-454d-b9d1-af3c7fce823a-kube-api-access-c5g7h" (OuterVolumeSpecName: "kube-api-access-c5g7h") pod "9dfe1397-45ce-454d-b9d1-af3c7fce823a" (UID: "9dfe1397-45ce-454d-b9d1-af3c7fce823a"). InnerVolumeSpecName "kube-api-access-c5g7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:08.010642 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.010608    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dfe1397-45ce-454d-b9d1-af3c7fce823a-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.010642 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.010636    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slhlp\" (UniqueName: \"kubernetes.io/projected/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-kube-api-access-slhlp\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.010642 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.010668    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dfe1397-45ce-454d-b9d1-af3c7fce823a-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.010642 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.010678    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5g7h\" (UniqueName: \"kubernetes.io/projected/9dfe1397-45ce-454d-b9d1-af3c7fce823a-kube-api-access-c5g7h\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.010926 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.010686    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.010926 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.010694    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ad4d68-5851-44e3-ab20-5228dccdfdc7-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.688164 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.688128    2578 generic.go:356] "Generic (PLEG): container finished" podID="c1ad4d68-5851-44e3-ab20-5228dccdfdc7" containerID="92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22" exitCode=0
Sep 11 18:41:08.688327 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.688172    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"
Sep 11 18:41:08.688327 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.688213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k" event={"ID":"c1ad4d68-5851-44e3-ab20-5228dccdfdc7","Type":"ContainerDied","Data":"92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22"}
Sep 11 18:41:08.688327 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.688267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k" event={"ID":"c1ad4d68-5851-44e3-ab20-5228dccdfdc7","Type":"ContainerDied","Data":"0e2addc299b4d0a94800ca8e8011e7bfbdef42e5f4bd9f289e33a2f1a47c77ff"}
Sep 11 18:41:08.688327 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.688288    2578 scope.go:117] "RemoveContainer" containerID="92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22"
Sep 11 18:41:08.689783 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.689489    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dfe1397-45ce-454d-b9d1-af3c7fce823a" containerID="50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1" exitCode=0
Sep 11 18:41:08.689783 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.689521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z" event={"ID":"9dfe1397-45ce-454d-b9d1-af3c7fce823a","Type":"ContainerDied","Data":"50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1"}
Sep 11 18:41:08.689783 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.689534    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"
Sep 11 18:41:08.689783 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.689553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z" event={"ID":"9dfe1397-45ce-454d-b9d1-af3c7fce823a","Type":"ContainerDied","Data":"6222fbd4e5ece9241ed60ce1452c2ae2f6c54d6f4aed4f4bbfe57c92bc9ce9df"}
Sep 11 18:41:08.697030 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.697010    2578 scope.go:117] "RemoveContainer" containerID="92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22"
Sep 11 18:41:08.697310 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:41:08.697290    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22\": container with ID starting with 92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22 not found: ID does not exist" containerID="92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22"
Sep 11 18:41:08.697383 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.697317    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22"} err="failed to get container status \"92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22\": rpc error: code = NotFound desc = could not find container \"92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22\": container with ID starting with 92b5043f651a17f822c8fdb24942881227cc6a8371650eb9dfbf9bed31715a22 not found: ID does not exist"
Sep 11 18:41:08.697383 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.697332    2578 scope.go:117] "RemoveContainer" containerID="50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1"
Sep 11 18:41:08.704409 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.704395    2578 scope.go:117] "RemoveContainer" containerID="50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1"
Sep 11 18:41:08.704678 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:41:08.704644    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1\": container with ID starting with 50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1 not found: ID does not exist" containerID="50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1"
Sep 11 18:41:08.704768 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.704679    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1"} err="failed to get container status \"50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1\": rpc error: code = NotFound desc = could not find container \"50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1\": container with ID starting with 50e61d4fc098cc04c84e04907715f7db943622c2fb185b612acd6beccdcee6e1 not found: ID does not exist"
Sep 11 18:41:08.708447 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.708427    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"]
Sep 11 18:41:08.716033 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.716013    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-047453/distinct-w0-047453-kserve-67c97f6676-msd6k"]
Sep 11 18:41:08.733455 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.733395    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"]
Sep 11 18:41:08.735355 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:08.735329    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-047453/premium-w0-047453-kserve-7bc46ff75c-r4z6z"]
Sep 11 18:41:09.938025 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:09.937986    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dfe1397-45ce-454d-b9d1-af3c7fce823a" path="/var/lib/kubelet/pods/9dfe1397-45ce-454d-b9d1-af3c7fce823a/volumes"
Sep 11 18:41:09.938569 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:09.938550    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1ad4d68-5851-44e3-ab20-5228dccdfdc7" path="/var/lib/kubelet/pods/c1ad4d68-5851-44e3-ab20-5228dccdfdc7/volumes"
Sep 11 18:41:22.834876 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.834829    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"]
Sep 11 18:41:22.835415 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.835398    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dfe1397-45ce-454d-b9d1-af3c7fce823a" containerName="main"
Sep 11 18:41:22.835496 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.835416    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dfe1397-45ce-454d-b9d1-af3c7fce823a" containerName="main"
Sep 11 18:41:22.835496 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.835438    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ad4d68-5851-44e3-ab20-5228dccdfdc7" containerName="main"
Sep 11 18:41:22.835496 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.835447    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ad4d68-5851-44e3-ab20-5228dccdfdc7" containerName="main"
Sep 11 18:41:22.835496 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.835476    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1053b8ae-1501-46f8-8b2f-b0a3488bd261" containerName="maas-api"
Sep 11 18:41:22.835496 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.835484    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1053b8ae-1501-46f8-8b2f-b0a3488bd261" containerName="maas-api"
Sep 11 18:41:22.835784 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.835574    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1053b8ae-1501-46f8-8b2f-b0a3488bd261" containerName="maas-api"
Sep 11 18:41:22.835784 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.835587    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1ad4d68-5851-44e3-ab20-5228dccdfdc7" containerName="main"
Sep 11 18:41:22.835784 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.835601    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dfe1397-45ce-454d-b9d1-af3c7fce823a" containerName="main"
Sep 11 18:41:22.839116 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.839096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:22.843098 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.843078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-553671-serving-cert\""
Sep 11 18:41:22.849867 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.849836    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"]
Sep 11 18:41:22.958184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.958151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:22.958348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.958187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea330cc-51eb-4a91-b987-1c70adf4e896-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:22.958348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.958263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vz47\" (UniqueName: \"kubernetes.io/projected/5ea330cc-51eb-4a91-b987-1c70adf4e896-kube-api-access-6vz47\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:22.958348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:22.958332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:23.059317 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.059275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:23.059317 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.059315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea330cc-51eb-4a91-b987-1c70adf4e896-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:23.059575 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.059431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vz47\" (UniqueName: \"kubernetes.io/projected/5ea330cc-51eb-4a91-b987-1c70adf4e896-kube-api-access-6vz47\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:23.059575 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.059481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:23.060091 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.060064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea330cc-51eb-4a91-b987-1c70adf4e896-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:23.061890 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.061868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:23.062014 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.061888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:23.068802 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.068775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vz47\" (UniqueName: \"kubernetes.io/projected/5ea330cc-51eb-4a91-b987-1c70adf4e896-kube-api-access-6vz47\") pod \"maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:23.153754 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.153675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:23.439170 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.439140    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"]
Sep 11 18:41:23.441061 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:41:23.441033    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea330cc_51eb_4a91_b987_1c70adf4e896.slice/crio-2c830ca6cd61231de7b804234fc563791b769843aefe488cd063b255a28e7d2a WatchSource:0}: Error finding container 2c830ca6cd61231de7b804234fc563791b769843aefe488cd063b255a28e7d2a: Status 404 returned error can't find the container with id 2c830ca6cd61231de7b804234fc563791b769843aefe488cd063b255a28e7d2a
Sep 11 18:41:23.442437 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.442415    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:41:23.583205 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.583174    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"]
Sep 11 18:41:23.587975 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.587953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:41:23.591639 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.591619    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-553671-dockercfg-kwmbm\""
Sep 11 18:41:23.591984 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.591968    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-553671\""
Sep 11 18:41:23.606103 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.606082    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"]
Sep 11 18:41:23.664871 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.664843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbcd7c45-75d7-43eb-a887-4eace477ec09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d\" (UID: \"bbcd7c45-75d7-43eb-a887-4eace477ec09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:41:23.665033 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.664982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhr9m\" (UniqueName: \"kubernetes.io/projected/bbcd7c45-75d7-43eb-a887-4eace477ec09-kube-api-access-nhr9m\") pod \"payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d\" (UID: \"bbcd7c45-75d7-43eb-a887-4eace477ec09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:41:23.765643 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.765553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbcd7c45-75d7-43eb-a887-4eace477ec09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d\" (UID: \"bbcd7c45-75d7-43eb-a887-4eace477ec09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:41:23.765834 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.765722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhr9m\" (UniqueName: \"kubernetes.io/projected/bbcd7c45-75d7-43eb-a887-4eace477ec09-kube-api-access-nhr9m\") pod \"payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d\" (UID: \"bbcd7c45-75d7-43eb-a887-4eace477ec09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:41:23.765912 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.765847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp" event={"ID":"5ea330cc-51eb-4a91-b987-1c70adf4e896","Type":"ContainerStarted","Data":"2c830ca6cd61231de7b804234fc563791b769843aefe488cd063b255a28e7d2a"}
Sep 11 18:41:23.766343 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.766324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbcd7c45-75d7-43eb-a887-4eace477ec09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d\" (UID: \"bbcd7c45-75d7-43eb-a887-4eace477ec09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:41:23.774444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.774421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhr9m\" (UniqueName: \"kubernetes.io/projected/bbcd7c45-75d7-43eb-a887-4eace477ec09-kube-api-access-nhr9m\") pod \"payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d\" (UID: \"bbcd7c45-75d7-43eb-a887-4eace477ec09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:41:23.898371 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:23.898339    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:41:24.170900 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:24.170875    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"]
Sep 11 18:41:24.172863 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:41:24.172832    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcd7c45_75d7_43eb_a887_4eace477ec09.slice/crio-86803151e9b28a784e14996f450ec1e9a34ad7d1ed79335ef8787711e2c0b25e WatchSource:0}: Error finding container 86803151e9b28a784e14996f450ec1e9a34ad7d1ed79335ef8787711e2c0b25e: Status 404 returned error can't find the container with id 86803151e9b28a784e14996f450ec1e9a34ad7d1ed79335ef8787711e2c0b25e
Sep 11 18:41:24.772500 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:24.772457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d" event={"ID":"bbcd7c45-75d7-43eb-a887-4eace477ec09","Type":"ContainerStarted","Data":"f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8"}
Sep 11 18:41:24.772726 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:24.772531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:41:24.772726 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:24.772552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d" event={"ID":"bbcd7c45-75d7-43eb-a887-4eace477ec09","Type":"ContainerStarted","Data":"86803151e9b28a784e14996f450ec1e9a34ad7d1ed79335ef8787711e2c0b25e"}
Sep 11 18:41:24.774116 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:24.774094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp" event={"ID":"5ea330cc-51eb-4a91-b987-1c70adf4e896","Type":"ContainerStarted","Data":"4debfd7714eee2b07beb90681112db67cdaf6b7eced2290ee1d66a76a54f3537"}
Sep 11 18:41:24.774237 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:24.774222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:24.792496 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:24.792448    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d" podStartSLOduration=1.792435048 podStartE2EDuration="1.792435048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:41:24.790732955 +0000 UTC m=+1549.369997516" watchObservedRunningTime="2026-09-11 18:41:24.792435048 +0000 UTC m=+1549.371699608"
Sep 11 18:41:24.812242 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:24.812196    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp" podStartSLOduration=2.4776057209999998 podStartE2EDuration="2.812182189s" totalImagesPullingTime="334.576468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:41:24.811851683 +0000 UTC m=+1549.391116243" watchObservedRunningTime="2026-09-11 18:41:24.812182189 +0000 UTC m=+1549.391446748"
Sep 11 18:41:30.785251 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:30.785213    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:41:35.781753 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:35.781718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:41:36.657815 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:36.657780    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"]
Sep 11 18:41:40.038765 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:40.038734    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"]
Sep 11 18:41:51.852343 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:51.852282    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87" podUID="fc41905c-ac79-4776-99d6-20ee7934b102" containerName="main" containerID="cri-o://829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443" gracePeriod=45
Sep 11 18:41:52.099561 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.099539    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:41:52.232395 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-model-cache\") pod \"fc41905c-ac79-4776-99d6-20ee7934b102\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") "
Sep 11 18:41:52.232395 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232359    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-home\") pod \"fc41905c-ac79-4776-99d6-20ee7934b102\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") "
Sep 11 18:41:52.232395 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232395    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-tmp-dir\") pod \"fc41905c-ac79-4776-99d6-20ee7934b102\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") "
Sep 11 18:41:52.232691 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232421    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd8lz\" (UniqueName: \"kubernetes.io/projected/fc41905c-ac79-4776-99d6-20ee7934b102-kube-api-access-wd8lz\") pod \"fc41905c-ac79-4776-99d6-20ee7934b102\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") "
Sep 11 18:41:52.232691 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc41905c-ac79-4776-99d6-20ee7934b102-tls-certs\") pod \"fc41905c-ac79-4776-99d6-20ee7934b102\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") "
Sep 11 18:41:52.232691 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232478    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-dshm\") pod \"fc41905c-ac79-4776-99d6-20ee7934b102\" (UID: \"fc41905c-ac79-4776-99d6-20ee7934b102\") "
Sep 11 18:41:52.232691 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232637    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc41905c-ac79-4776-99d6-20ee7934b102" (UID: "fc41905c-ac79-4776-99d6-20ee7934b102"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.232691 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-home" (OuterVolumeSpecName: "home") pod "fc41905c-ac79-4776-99d6-20ee7934b102" (UID: "fc41905c-ac79-4776-99d6-20ee7934b102"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.232909 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232695    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc41905c-ac79-4776-99d6-20ee7934b102" (UID: "fc41905c-ac79-4776-99d6-20ee7934b102"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.232962 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232937    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.232962 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232950    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.232962 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.232958    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.234624 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.234596    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc41905c-ac79-4776-99d6-20ee7934b102-kube-api-access-wd8lz" (OuterVolumeSpecName: "kube-api-access-wd8lz") pod "fc41905c-ac79-4776-99d6-20ee7934b102" (UID: "fc41905c-ac79-4776-99d6-20ee7934b102"). InnerVolumeSpecName "kube-api-access-wd8lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:52.234624 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.234610    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc41905c-ac79-4776-99d6-20ee7934b102-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc41905c-ac79-4776-99d6-20ee7934b102" (UID: "fc41905c-ac79-4776-99d6-20ee7934b102"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:52.234785 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.234718    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-dshm" (OuterVolumeSpecName: "dshm") pod "fc41905c-ac79-4776-99d6-20ee7934b102" (UID: "fc41905c-ac79-4776-99d6-20ee7934b102"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.334294 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.334272    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd8lz\" (UniqueName: \"kubernetes.io/projected/fc41905c-ac79-4776-99d6-20ee7934b102-kube-api-access-wd8lz\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.334294 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.334296    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc41905c-ac79-4776-99d6-20ee7934b102-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.334444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.334305    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc41905c-ac79-4776-99d6-20ee7934b102-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.912493 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.912456    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc41905c-ac79-4776-99d6-20ee7934b102" containerID="829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443" exitCode=0
Sep 11 18:41:52.912493 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.912493    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"
Sep 11 18:41:52.912998 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.912535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87" event={"ID":"fc41905c-ac79-4776-99d6-20ee7934b102","Type":"ContainerDied","Data":"829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443"}
Sep 11 18:41:52.912998 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.912577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87" event={"ID":"fc41905c-ac79-4776-99d6-20ee7934b102","Type":"ContainerDied","Data":"02892a1431c415d797e3e29877256bdfcd1ac6f1e13f6318415cbfa9a20136d7"}
Sep 11 18:41:52.912998 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.912596    2578 scope.go:117] "RemoveContainer" containerID="829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443"
Sep 11 18:41:52.922012 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.921996    2578 scope.go:117] "RemoveContainer" containerID="829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443"
Sep 11 18:41:52.922247 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:41:52.922229    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443\": container with ID starting with 829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443 not found: ID does not exist" containerID="829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443"
Sep 11 18:41:52.922295 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.922255    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443"} err="failed to get container status \"829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443\": rpc error: code = NotFound desc = could not find container \"829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443\": container with ID starting with 829e5b05cf2272958d499bad1c4b65dd9ba9d28457f1207d24c7cd5d6ecd1443 not found: ID does not exist"
Sep 11 18:41:52.933840 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.933816    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"]
Sep 11 18:41:52.940151 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:52.940133    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-b-ccb0c59a-kserve-8f7f6d9c7-gww87"]
Sep 11 18:41:53.937032 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:53.936999    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc41905c-ac79-4776-99d6-20ee7934b102" path="/var/lib/kubelet/pods/fc41905c-ac79-4776-99d6-20ee7934b102/volumes"
Sep 11 18:41:55.057691 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.057614    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5" podUID="a72fdabb-2810-4e4c-98e0-b7fb56b0613d" containerName="main" containerID="cri-o://a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8" gracePeriod=45
Sep 11 18:41:55.301949 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.301923    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:41:55.359499 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.359439    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-home\") pod \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") "
Sep 11 18:41:55.359499 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.359469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tmp-dir\") pod \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") "
Sep 11 18:41:55.359729 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.359540    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2twxn\" (UniqueName: \"kubernetes.io/projected/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-kube-api-access-2twxn\") pod \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") "
Sep 11 18:41:55.359729 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.359560    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-model-cache\") pod \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") "
Sep 11 18:41:55.359729 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.359615    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tls-certs\") pod \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") "
Sep 11 18:41:55.359729 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.359672    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-dshm\") pod \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\" (UID: \"a72fdabb-2810-4e4c-98e0-b7fb56b0613d\") "
Sep 11 18:41:55.359916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.359731    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-home" (OuterVolumeSpecName: "home") pod "a72fdabb-2810-4e4c-98e0-b7fb56b0613d" (UID: "a72fdabb-2810-4e4c-98e0-b7fb56b0613d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:55.359916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.359809    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a72fdabb-2810-4e4c-98e0-b7fb56b0613d" (UID: "a72fdabb-2810-4e4c-98e0-b7fb56b0613d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:55.359916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.359887    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a72fdabb-2810-4e4c-98e0-b7fb56b0613d" (UID: "a72fdabb-2810-4e4c-98e0-b7fb56b0613d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:55.360013 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.359995    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.360046 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.360016    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.360046 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.360033    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.361573 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.361548    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-kube-api-access-2twxn" (OuterVolumeSpecName: "kube-api-access-2twxn") pod "a72fdabb-2810-4e4c-98e0-b7fb56b0613d" (UID: "a72fdabb-2810-4e4c-98e0-b7fb56b0613d"). InnerVolumeSpecName "kube-api-access-2twxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:55.361696 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.361584    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-dshm" (OuterVolumeSpecName: "dshm") pod "a72fdabb-2810-4e4c-98e0-b7fb56b0613d" (UID: "a72fdabb-2810-4e4c-98e0-b7fb56b0613d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:55.361696 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.361627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a72fdabb-2810-4e4c-98e0-b7fb56b0613d" (UID: "a72fdabb-2810-4e4c-98e0-b7fb56b0613d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:55.460975 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.460944    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.460975 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.460970    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.461122 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.460982    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2twxn\" (UniqueName: \"kubernetes.io/projected/a72fdabb-2810-4e4c-98e0-b7fb56b0613d-kube-api-access-2twxn\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.929672 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.929621    2578 generic.go:356] "Generic (PLEG): container finished" podID="a72fdabb-2810-4e4c-98e0-b7fb56b0613d" containerID="a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8" exitCode=0
Sep 11 18:41:55.929833 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.929689    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"
Sep 11 18:41:55.929833 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.929707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5" event={"ID":"a72fdabb-2810-4e4c-98e0-b7fb56b0613d","Type":"ContainerDied","Data":"a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8"}
Sep 11 18:41:55.929833 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.929753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5" event={"ID":"a72fdabb-2810-4e4c-98e0-b7fb56b0613d","Type":"ContainerDied","Data":"9cfe340be3e5eddf07fa32bc9c697c1fbfc071708218ef9737048617f151da24"}
Sep 11 18:41:55.929833 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.929771    2578 scope.go:117] "RemoveContainer" containerID="a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8"
Sep 11 18:41:55.939707 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.939691    2578 scope.go:117] "RemoveContainer" containerID="a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8"
Sep 11 18:41:55.939980 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:41:55.939960    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8\": container with ID starting with a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8 not found: ID does not exist" containerID="a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8"
Sep 11 18:41:55.940050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.939987    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8"} err="failed to get container status \"a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8\": rpc error: code = NotFound desc = could not find container \"a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8\": container with ID starting with a02c59bf8e9d02f32d1b06c6f91ec51993462820c09aa4534698ae601c63d7d8 not found: ID does not exist"
Sep 11 18:41:55.951495 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.951474    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"]
Sep 11 18:41:55.955793 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:55.955773    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/e2e-conflict-a-ccb0c59a-kserve-676c447c87-426l5"]
Sep 11 18:41:56.650337 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.650306    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"]
Sep 11 18:41:56.650923 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.650904    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc41905c-ac79-4776-99d6-20ee7934b102" containerName="main"
Sep 11 18:41:56.651011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.650925    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc41905c-ac79-4776-99d6-20ee7934b102" containerName="main"
Sep 11 18:41:56.651011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.650945    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a72fdabb-2810-4e4c-98e0-b7fb56b0613d" containerName="main"
Sep 11 18:41:56.651011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.650954    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a72fdabb-2810-4e4c-98e0-b7fb56b0613d" containerName="main"
Sep 11 18:41:56.651162 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.651048    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a72fdabb-2810-4e4c-98e0-b7fb56b0613d" containerName="main"
Sep 11 18:41:56.651162 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.651065    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc41905c-ac79-4776-99d6-20ee7934b102" containerName="main"
Sep 11 18:41:56.655758 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.655736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.658542 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.658524    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-200c07\"/\"distinct-w4-200c07-kserve-self-signed-certs\""
Sep 11 18:41:56.663012 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.662988    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"]
Sep 11 18:41:56.770469 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.770436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-model-cache\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.770602 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.770480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-tmp-dir\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.770602 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.770508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1261164-4a5b-48df-b1e4-8563bde8a2be-tls-certs\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.770602 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.770576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-home\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.770736 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.770613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj6kx\" (UniqueName: \"kubernetes.io/projected/f1261164-4a5b-48df-b1e4-8563bde8a2be-kube-api-access-rj6kx\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.770736 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.770680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-dshm\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.871766 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.871739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-dshm\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.871891 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.871794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-model-cache\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.871891 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.871822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-tmp-dir\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.871891 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.871844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1261164-4a5b-48df-b1e4-8563bde8a2be-tls-certs\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.871891 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.871867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-home\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.871891 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.871884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6kx\" (UniqueName: \"kubernetes.io/projected/f1261164-4a5b-48df-b1e4-8563bde8a2be-kube-api-access-rj6kx\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.872236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.872208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-model-cache\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.872298 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.872218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-tmp-dir\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.872298 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.872253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-home\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.873902 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.873884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-dshm\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.874220 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.874199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1261164-4a5b-48df-b1e4-8563bde8a2be-tls-certs\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.880208 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.880189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj6kx\" (UniqueName: \"kubernetes.io/projected/f1261164-4a5b-48df-b1e4-8563bde8a2be-kube-api-access-rj6kx\") pod \"distinct-w4-200c07-kserve-784d7f8c57-b4855\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:56.966328 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:56.966257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:41:57.238380 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:57.238353    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"]
Sep 11 18:41:57.239796 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:41:57.239766    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1261164_4a5b_48df_b1e4_8563bde8a2be.slice/crio-f173922e6f48ad7a2e0bca0fa3bd652703992d64aa13ac5c76cbe2803d57a825 WatchSource:0}: Error finding container f173922e6f48ad7a2e0bca0fa3bd652703992d64aa13ac5c76cbe2803d57a825: Status 404 returned error can't find the container with id f173922e6f48ad7a2e0bca0fa3bd652703992d64aa13ac5c76cbe2803d57a825
Sep 11 18:41:57.937847 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:57.937819    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a72fdabb-2810-4e4c-98e0-b7fb56b0613d" path="/var/lib/kubelet/pods/a72fdabb-2810-4e4c-98e0-b7fb56b0613d/volumes"
Sep 11 18:41:57.940938 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:57.940909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855" event={"ID":"f1261164-4a5b-48df-b1e4-8563bde8a2be","Type":"ContainerStarted","Data":"f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532"}
Sep 11 18:41:57.941056 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:57.940945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855" event={"ID":"f1261164-4a5b-48df-b1e4-8563bde8a2be","Type":"ContainerStarted","Data":"f173922e6f48ad7a2e0bca0fa3bd652703992d64aa13ac5c76cbe2803d57a825"}
Sep 11 18:41:57.959717 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:41:57.959676    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855" podStartSLOduration=1.959662756 podStartE2EDuration="1.959662756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:41:57.957706807 +0000 UTC m=+1582.536971367" watchObservedRunningTime="2026-09-11 18:41:57.959662756 +0000 UTC m=+1582.538927310"
Sep 11 18:42:06.385849 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.385819    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"]
Sep 11 18:42:06.386389 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.386052    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8" podUID="ac944916-4442-4528-941a-63d7f4c2fb78" containerName="payload-pre-processing" containerID="cri-o://fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9" gracePeriod=30
Sep 11 18:42:06.631837 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.631812    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:42:06.753928 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.753861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz9ln\" (UniqueName: \"kubernetes.io/projected/ac944916-4442-4528-941a-63d7f4c2fb78-kube-api-access-mz9ln\") pod \"ac944916-4442-4528-941a-63d7f4c2fb78\" (UID: \"ac944916-4442-4528-941a-63d7f4c2fb78\") "
Sep 11 18:42:06.754079 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.753965    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac944916-4442-4528-941a-63d7f4c2fb78-plugins-config-volume\") pod \"ac944916-4442-4528-941a-63d7f4c2fb78\" (UID: \"ac944916-4442-4528-941a-63d7f4c2fb78\") "
Sep 11 18:42:06.754294 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.754273    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac944916-4442-4528-941a-63d7f4c2fb78-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac944916-4442-4528-941a-63d7f4c2fb78" (UID: "ac944916-4442-4528-941a-63d7f4c2fb78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:42:06.755888 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.755858    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac944916-4442-4528-941a-63d7f4c2fb78-kube-api-access-mz9ln" (OuterVolumeSpecName: "kube-api-access-mz9ln") pod "ac944916-4442-4528-941a-63d7f4c2fb78" (UID: "ac944916-4442-4528-941a-63d7f4c2fb78"). InnerVolumeSpecName "kube-api-access-mz9ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:06.854887 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.854856    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz9ln\" (UniqueName: \"kubernetes.io/projected/ac944916-4442-4528-941a-63d7f4c2fb78-kube-api-access-mz9ln\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:42:06.854887 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.854883    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac944916-4442-4528-941a-63d7f4c2fb78-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:42:06.966882 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.966846    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:42:06.967011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.966891    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:42:06.979203 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.979186    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:42:06.984400 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.984378    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac944916-4442-4528-941a-63d7f4c2fb78" containerID="fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9" exitCode=0
Sep 11 18:42:06.984480 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.984414    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"
Sep 11 18:42:06.984480 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.984442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8" event={"ID":"ac944916-4442-4528-941a-63d7f4c2fb78","Type":"ContainerDied","Data":"fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9"}
Sep 11 18:42:06.984480 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.984476    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8" event={"ID":"ac944916-4442-4528-941a-63d7f4c2fb78","Type":"ContainerDied","Data":"a20d9362350c35e74303a65cb1ac32dec07cec78c4abd27ffba82be867d58f74"}
Sep 11 18:42:06.984580 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.984497    2578 scope.go:117] "RemoveContainer" containerID="fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9"
Sep 11 18:42:06.996408 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.996386    2578 scope.go:117] "RemoveContainer" containerID="fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9"
Sep 11 18:42:06.997152 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:42:06.997128    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9\": container with ID starting with fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9 not found: ID does not exist" containerID="fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9"
Sep 11 18:42:06.997226 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.997163    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9"} err="failed to get container status \"fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9\": rpc error: code = NotFound desc = could not find container \"fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9\": container with ID starting with fd26e92797f076f1e006139a5372b876af0fc9099247b5e1858bc296430091e9 not found: ID does not exist"
Sep 11 18:42:06.998716 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:06.998700    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:42:07.038476 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:07.038452    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"]
Sep 11 18:42:07.044218 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:07.044196    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-699caf96-fd74cc7bb-8pvt8"]
Sep 11 18:42:07.937374 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:42:07.937340    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac944916-4442-4528-941a-63d7f4c2fb78" path="/var/lib/kubelet/pods/ac944916-4442-4528-941a-63d7f4c2fb78/volumes"
Sep 11 18:43:26.352214 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.352175    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"]
Sep 11 18:43:26.352842 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.352822    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac944916-4442-4528-941a-63d7f4c2fb78" containerName="payload-pre-processing"
Sep 11 18:43:26.352894 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.352846    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac944916-4442-4528-941a-63d7f4c2fb78" containerName="payload-pre-processing"
Sep 11 18:43:26.352956 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.352943    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac944916-4442-4528-941a-63d7f4c2fb78" containerName="payload-pre-processing"
Sep 11 18:43:26.356553 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.356529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.359628 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.359604    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-200c07\"/\"distinct-2-w4-200c07-kserve-self-signed-certs\""
Sep 11 18:43:26.365244 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.365222    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"]
Sep 11 18:43:26.455337 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.455279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cce2f0-187d-4593-be87-c8441b9dc9b6-tls-certs\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.455337 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.455345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-model-cache\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.455596 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.455429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5468z\" (UniqueName: \"kubernetes.io/projected/22cce2f0-187d-4593-be87-c8441b9dc9b6-kube-api-access-5468z\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.455596 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.455471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-home\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.455596 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.455531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-dshm\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.455596 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.455581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-tmp-dir\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.556734 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.556679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5468z\" (UniqueName: \"kubernetes.io/projected/22cce2f0-187d-4593-be87-c8441b9dc9b6-kube-api-access-5468z\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.556734 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.556748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-home\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.556991 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.556801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-dshm\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.556991 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.556840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-tmp-dir\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.556991 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.556881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cce2f0-187d-4593-be87-c8441b9dc9b6-tls-certs\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.556991 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.556919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-model-cache\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.557194 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.557174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-home\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.557302 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.557281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-tmp-dir\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.557418 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.557301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-model-cache\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.559128 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.559108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-dshm\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.559222 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.559205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cce2f0-187d-4593-be87-c8441b9dc9b6-tls-certs\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.573169 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.573136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5468z\" (UniqueName: \"kubernetes.io/projected/22cce2f0-187d-4593-be87-c8441b9dc9b6-kube-api-access-5468z\") pod \"distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") " pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.666908 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.666813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:26.941205 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:26.941122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"]
Sep 11 18:43:26.945227 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:43:26.945198    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22cce2f0_187d_4593_be87_c8441b9dc9b6.slice/crio-21ab7d20cb62e18995e82d2cc54f94e43864d0d105e144ecf73c8eda8d7fd4e8 WatchSource:0}: Error finding container 21ab7d20cb62e18995e82d2cc54f94e43864d0d105e144ecf73c8eda8d7fd4e8: Status 404 returned error can't find the container with id 21ab7d20cb62e18995e82d2cc54f94e43864d0d105e144ecf73c8eda8d7fd4e8
Sep 11 18:43:27.363194 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:27.363160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr" event={"ID":"22cce2f0-187d-4593-be87-c8441b9dc9b6","Type":"ContainerStarted","Data":"bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02"}
Sep 11 18:43:27.363586 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:27.363201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr" event={"ID":"22cce2f0-187d-4593-be87-c8441b9dc9b6","Type":"ContainerStarted","Data":"21ab7d20cb62e18995e82d2cc54f94e43864d0d105e144ecf73c8eda8d7fd4e8"}
Sep 11 18:43:27.388418 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:27.388362    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr" podStartSLOduration=1.38834567 podStartE2EDuration="1.38834567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:43:27.388100313 +0000 UTC m=+1671.967364873" watchObservedRunningTime="2026-09-11 18:43:27.38834567 +0000 UTC m=+1671.967610230"
Sep 11 18:43:36.667620 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:36.667582    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:36.668020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:36.667633    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:36.679621 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:36.679597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:43:37.424456 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:43:37.424428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:44:16.146917 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.146883    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"]
Sep 11 18:44:16.150334 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.150317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:44:16.153360 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.153338    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-883e85\""
Sep 11 18:44:16.153490 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.153408    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-883e85-dockercfg-bbtgq\""
Sep 11 18:44:16.162409 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.162388    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"]
Sep 11 18:44:16.215882 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.215854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkz46\" (UniqueName: \"kubernetes.io/projected/d50e9b73-73fb-4700-a7a2-0c36000188ac-kube-api-access-pkz46\") pod \"payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm\" (UID: \"d50e9b73-73fb-4700-a7a2-0c36000188ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:44:16.215992 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.215919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50e9b73-73fb-4700-a7a2-0c36000188ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm\" (UID: \"d50e9b73-73fb-4700-a7a2-0c36000188ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:44:16.218753 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.218732    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"]
Sep 11 18:44:16.222108 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.222094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:44:16.233379 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.233360    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"]
Sep 11 18:44:16.316356 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.316325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkz46\" (UniqueName: \"kubernetes.io/projected/d50e9b73-73fb-4700-a7a2-0c36000188ac-kube-api-access-pkz46\") pod \"payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm\" (UID: \"d50e9b73-73fb-4700-a7a2-0c36000188ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:44:16.316514 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.316379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197ccb8d-73cf-41d6-a18c-d263d7c53700-plugins-config-volume\") pod \"payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9\" (UID: \"197ccb8d-73cf-41d6-a18c-d263d7c53700\") " pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:44:16.316514 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.316414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkfv\" (UniqueName: \"kubernetes.io/projected/197ccb8d-73cf-41d6-a18c-d263d7c53700-kube-api-access-jgkfv\") pod \"payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9\" (UID: \"197ccb8d-73cf-41d6-a18c-d263d7c53700\") " pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:44:16.316514 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.316445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50e9b73-73fb-4700-a7a2-0c36000188ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm\" (UID: \"d50e9b73-73fb-4700-a7a2-0c36000188ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:44:16.317107 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.317084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50e9b73-73fb-4700-a7a2-0c36000188ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm\" (UID: \"d50e9b73-73fb-4700-a7a2-0c36000188ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:44:16.325313 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.325288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkz46\" (UniqueName: \"kubernetes.io/projected/d50e9b73-73fb-4700-a7a2-0c36000188ac-kube-api-access-pkz46\") pod \"payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm\" (UID: \"d50e9b73-73fb-4700-a7a2-0c36000188ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:44:16.417143 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.417085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197ccb8d-73cf-41d6-a18c-d263d7c53700-plugins-config-volume\") pod \"payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9\" (UID: \"197ccb8d-73cf-41d6-a18c-d263d7c53700\") " pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:44:16.417143 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.417124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkfv\" (UniqueName: \"kubernetes.io/projected/197ccb8d-73cf-41d6-a18c-d263d7c53700-kube-api-access-jgkfv\") pod \"payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9\" (UID: \"197ccb8d-73cf-41d6-a18c-d263d7c53700\") " pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:44:16.417744 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.417725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197ccb8d-73cf-41d6-a18c-d263d7c53700-plugins-config-volume\") pod \"payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9\" (UID: \"197ccb8d-73cf-41d6-a18c-d263d7c53700\") " pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:44:16.425850 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.425824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkfv\" (UniqueName: \"kubernetes.io/projected/197ccb8d-73cf-41d6-a18c-d263d7c53700-kube-api-access-jgkfv\") pod \"payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9\" (UID: \"197ccb8d-73cf-41d6-a18c-d263d7c53700\") " pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:44:16.461068 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.461042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:44:16.534925 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.534898    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:44:16.731474 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.731451    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"]
Sep 11 18:44:16.733980 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:44:16.733957    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50e9b73_73fb_4700_a7a2_0c36000188ac.slice/crio-7d8735114b6de8686f644fcdfe089762fb87dd7442e9592285f4132d3f5d89d9 WatchSource:0}: Error finding container 7d8735114b6de8686f644fcdfe089762fb87dd7442e9592285f4132d3f5d89d9: Status 404 returned error can't find the container with id 7d8735114b6de8686f644fcdfe089762fb87dd7442e9592285f4132d3f5d89d9
Sep 11 18:44:16.819680 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:16.819638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"]
Sep 11 18:44:16.822238 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:44:16.822209    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197ccb8d_73cf_41d6_a18c_d263d7c53700.slice/crio-6597d94209d2566c5ff62c81bff3ce5d9a37f80586f91e1e15bc542f32c1240b WatchSource:0}: Error finding container 6597d94209d2566c5ff62c81bff3ce5d9a37f80586f91e1e15bc542f32c1240b: Status 404 returned error can't find the container with id 6597d94209d2566c5ff62c81bff3ce5d9a37f80586f91e1e15bc542f32c1240b
Sep 11 18:44:17.610726 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:17.610689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm" event={"ID":"d50e9b73-73fb-4700-a7a2-0c36000188ac","Type":"ContainerStarted","Data":"2ea0c0d109a999b8b012c8cf9e7b7de0b6a8a97019e4688a925cc17b077ad867"}
Sep 11 18:44:17.611169 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:17.610760    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:44:17.611169 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:17.610784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm" event={"ID":"d50e9b73-73fb-4700-a7a2-0c36000188ac","Type":"ContainerStarted","Data":"7d8735114b6de8686f644fcdfe089762fb87dd7442e9592285f4132d3f5d89d9"}
Sep 11 18:44:17.612273 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:17.612250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9" event={"ID":"197ccb8d-73cf-41d6-a18c-d263d7c53700","Type":"ContainerStarted","Data":"bacd3335fad2fc98b6fb70771309abb7784fe2df602954b7c4007dbedda56dc9"}
Sep 11 18:44:17.612557 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:17.612293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9" event={"ID":"197ccb8d-73cf-41d6-a18c-d263d7c53700","Type":"ContainerStarted","Data":"6597d94209d2566c5ff62c81bff3ce5d9a37f80586f91e1e15bc542f32c1240b"}
Sep 11 18:44:17.612557 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:17.612322    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:44:17.630997 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:17.630944    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm" podStartSLOduration=1.630926808 podStartE2EDuration="1.630926808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:17.629051204 +0000 UTC m=+1722.208315765" watchObservedRunningTime="2026-09-11 18:44:17.630926808 +0000 UTC m=+1722.210191369"
Sep 11 18:44:17.647214 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:17.647160    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9" podStartSLOduration=1.647141381 podStartE2EDuration="1.647141381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:17.645103231 +0000 UTC m=+1722.224367792" watchObservedRunningTime="2026-09-11 18:44:17.647141381 +0000 UTC m=+1722.226405941"
Sep 11 18:44:28.619260 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:28.619226    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:44:28.619723 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:28.619402    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:44:51.054597 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.054562    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"]
Sep 11 18:44:51.058272 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.058255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:44:51.060994 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.060975    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5cbc66\""
Sep 11 18:44:51.068628 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.068602    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"]
Sep 11 18:44:51.141944 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.141908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979040e9-ca5c-4741-aebc-2bd65e1c7b60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5cbc66-65879468d7-968wb\" (UID: \"979040e9-ca5c-4741-aebc-2bd65e1c7b60\") " pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:44:51.141944 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.141943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtv5\" (UniqueName: \"kubernetes.io/projected/979040e9-ca5c-4741-aebc-2bd65e1c7b60-kube-api-access-pgtv5\") pod \"payload-processing-e2e-api-b-5cbc66-65879468d7-968wb\" (UID: \"979040e9-ca5c-4741-aebc-2bd65e1c7b60\") " pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:44:51.243414 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.243378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979040e9-ca5c-4741-aebc-2bd65e1c7b60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5cbc66-65879468d7-968wb\" (UID: \"979040e9-ca5c-4741-aebc-2bd65e1c7b60\") " pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:44:51.243414 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.243415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtv5\" (UniqueName: \"kubernetes.io/projected/979040e9-ca5c-4741-aebc-2bd65e1c7b60-kube-api-access-pgtv5\") pod \"payload-processing-e2e-api-b-5cbc66-65879468d7-968wb\" (UID: \"979040e9-ca5c-4741-aebc-2bd65e1c7b60\") " pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:44:51.244101 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.244080    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979040e9-ca5c-4741-aebc-2bd65e1c7b60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5cbc66-65879468d7-968wb\" (UID: \"979040e9-ca5c-4741-aebc-2bd65e1c7b60\") " pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:44:51.253330 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.253297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtv5\" (UniqueName: \"kubernetes.io/projected/979040e9-ca5c-4741-aebc-2bd65e1c7b60-kube-api-access-pgtv5\") pod \"payload-processing-e2e-api-b-5cbc66-65879468d7-968wb\" (UID: \"979040e9-ca5c-4741-aebc-2bd65e1c7b60\") " pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:44:51.368155 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.368058    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:44:51.641059 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.641017    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"]
Sep 11 18:44:51.644113 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:44:51.644070    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979040e9_ca5c_4741_aebc_2bd65e1c7b60.slice/crio-aa32c6cec2d3a52adf95206d6fe173c37ce1f96907231dafa89753a947cd4af4 WatchSource:0}: Error finding container aa32c6cec2d3a52adf95206d6fe173c37ce1f96907231dafa89753a947cd4af4: Status 404 returned error can't find the container with id aa32c6cec2d3a52adf95206d6fe173c37ce1f96907231dafa89753a947cd4af4
Sep 11 18:44:51.785471 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.785435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb" event={"ID":"979040e9-ca5c-4741-aebc-2bd65e1c7b60","Type":"ContainerStarted","Data":"9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531"}
Sep 11 18:44:51.785689 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.785503    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:44:51.785689 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.785525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb" event={"ID":"979040e9-ca5c-4741-aebc-2bd65e1c7b60","Type":"ContainerStarted","Data":"aa32c6cec2d3a52adf95206d6fe173c37ce1f96907231dafa89753a947cd4af4"}
Sep 11 18:44:51.801843 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:44:51.801786    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb" podStartSLOduration=0.801768535 podStartE2EDuration="801.768535ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:51.801277837 +0000 UTC m=+1756.380542397" watchObservedRunningTime="2026-09-11 18:44:51.801768535 +0000 UTC m=+1756.381033094"
Sep 11 18:45:00.136244 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.136191    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh"]
Sep 11 18:45:00.139881 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.139861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh"
Sep 11 18:45:00.142708 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.142679    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmfp\""
Sep 11 18:45:00.150015 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.149989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh"
Sep 11 18:45:00.150801 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.150771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh"]
Sep 11 18:45:00.163811 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.163763    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr"]
Sep 11 18:45:00.167900 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.167875    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb"]
Sep 11 18:45:00.168038 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.168009    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr"
Sep 11 18:45:00.171986 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.171958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb"
Sep 11 18:45:00.184031 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.183954    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr"]
Sep 11 18:45:00.187183 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.187135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb"]
Sep 11 18:45:00.195997 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.195788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr"
Sep 11 18:45:00.205117 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.205087    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb"
Sep 11 18:45:00.438095 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.438070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh"]
Sep 11 18:45:00.440171 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:45:00.440141    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4874b1_2d71_4b9a_b6e6_bbea4ea9d31e.slice/crio-42f26b0f1fff3e1bd907fa0c10b6a97a9e479271ec593f170376f732c65c2335 WatchSource:0}: Error finding container 42f26b0f1fff3e1bd907fa0c10b6a97a9e479271ec593f170376f732c65c2335: Status 404 returned error can't find the container with id 42f26b0f1fff3e1bd907fa0c10b6a97a9e479271ec593f170376f732c65c2335
Sep 11 18:45:00.489812 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.489786    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr"]
Sep 11 18:45:00.491395 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:45:00.491359    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a5f3c0_6420_4606_9b39_ec06abb1cc48.slice/crio-1c3ee34c3bd3fea9520daeffab976ffe32d9dd3c9b0c0caad6a3e4c95cd54238 WatchSource:0}: Error finding container 1c3ee34c3bd3fea9520daeffab976ffe32d9dd3c9b0c0caad6a3e4c95cd54238: Status 404 returned error can't find the container with id 1c3ee34c3bd3fea9520daeffab976ffe32d9dd3c9b0c0caad6a3e4c95cd54238
Sep 11 18:45:00.503346 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.503323    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb"]
Sep 11 18:45:00.505336 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:45:00.505310    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8530cdd1_b251_4b14_93ad_46fe3695d58c.slice/crio-69891518a9fada584ba50bd67e26ddd83d30119d7843d2a7a675261b681ebbd4 WatchSource:0}: Error finding container 69891518a9fada584ba50bd67e26ddd83d30119d7843d2a7a675261b681ebbd4: Status 404 returned error can't find the container with id 69891518a9fada584ba50bd67e26ddd83d30119d7843d2a7a675261b681ebbd4
Sep 11 18:45:00.836969 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.836925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb" event={"ID":"8530cdd1-b251-4b14-93ad-46fe3695d58c","Type":"ContainerStarted","Data":"69891518a9fada584ba50bd67e26ddd83d30119d7843d2a7a675261b681ebbd4"}
Sep 11 18:45:00.838357 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.838322    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh" event={"ID":"bd4874b1-2d71-4b9a-b6e6-bbea4ea9d31e","Type":"ContainerStarted","Data":"42f26b0f1fff3e1bd907fa0c10b6a97a9e479271ec593f170376f732c65c2335"}
Sep 11 18:45:00.839796 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:00.839770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr" event={"ID":"52a5f3c0-6420-4606-9b39-ec06abb1cc48","Type":"ContainerStarted","Data":"1c3ee34c3bd3fea9520daeffab976ffe32d9dd3c9b0c0caad6a3e4c95cd54238"}
Sep 11 18:45:02.793832 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:02.793800    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:45:03.862278 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:03.862237    2578 generic.go:356] "Generic (PLEG): container finished" podID="bd4874b1-2d71-4b9a-b6e6-bbea4ea9d31e" containerID="4766d1e79b1d1c6eb7482eade56204d39200d29db3b34924ca07b3a8b82d15f5" exitCode=0
Sep 11 18:45:03.862743 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:03.862351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh" event={"ID":"bd4874b1-2d71-4b9a-b6e6-bbea4ea9d31e","Type":"ContainerDied","Data":"4766d1e79b1d1c6eb7482eade56204d39200d29db3b34924ca07b3a8b82d15f5"}
Sep 11 18:45:03.863988 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:03.863959    2578 generic.go:356] "Generic (PLEG): container finished" podID="52a5f3c0-6420-4606-9b39-ec06abb1cc48" containerID="fe91dab2f5e9a819b76b32f1d826196b3ad3bb3fde28668d81b618332974ca6d" exitCode=0
Sep 11 18:45:03.864124 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:03.864037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr" event={"ID":"52a5f3c0-6420-4606-9b39-ec06abb1cc48","Type":"ContainerDied","Data":"fe91dab2f5e9a819b76b32f1d826196b3ad3bb3fde28668d81b618332974ca6d"}
Sep 11 18:45:05.001188 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:05.001158    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh"
Sep 11 18:45:05.046765 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:05.046729    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr"
Sep 11 18:45:05.876087 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:05.876051    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh"
Sep 11 18:45:05.876296 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:05.876057    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh" event={"ID":"bd4874b1-2d71-4b9a-b6e6-bbea4ea9d31e","Type":"ContainerDied","Data":"42f26b0f1fff3e1bd907fa0c10b6a97a9e479271ec593f170376f732c65c2335"}
Sep 11 18:45:05.876296 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:05.876196    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f26b0f1fff3e1bd907fa0c10b6a97a9e479271ec593f170376f732c65c2335"
Sep 11 18:45:05.877401 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:05.877373    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr"
Sep 11 18:45:05.877540 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:05.877402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr" event={"ID":"52a5f3c0-6420-4606-9b39-ec06abb1cc48","Type":"ContainerDied","Data":"1c3ee34c3bd3fea9520daeffab976ffe32d9dd3c9b0c0caad6a3e4c95cd54238"}
Sep 11 18:45:05.877540 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:05.877435    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c3ee34c3bd3fea9520daeffab976ffe32d9dd3c9b0c0caad6a3e4c95cd54238"
Sep 11 18:45:18.948253 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:18.948141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb" event={"ID":"8530cdd1-b251-4b14-93ad-46fe3695d58c","Type":"ContainerStarted","Data":"bac98ce7259dbf87465b818b1fe6d9bd46f40d636f2b9a66d117dfca94044260"}
Sep 11 18:45:18.967749 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:18.967695    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb" podStartSLOduration=1.140673548 podStartE2EDuration="18.967680089s" totalImagesPullingTime="17.827006541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:18.96729471 +0000 UTC m=+1783.546559270" watchObservedRunningTime="2026-09-11 18:45:18.967680089 +0000 UTC m=+1783.546944646"
Sep 11 18:45:19.954516 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:19.954477    2578 generic.go:356] "Generic (PLEG): container finished" podID="8530cdd1-b251-4b14-93ad-46fe3695d58c" containerID="bac98ce7259dbf87465b818b1fe6d9bd46f40d636f2b9a66d117dfca94044260" exitCode=0
Sep 11 18:45:19.954940 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:19.954560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb" event={"ID":"8530cdd1-b251-4b14-93ad-46fe3695d58c","Type":"ContainerDied","Data":"bac98ce7259dbf87465b818b1fe6d9bd46f40d636f2b9a66d117dfca94044260"}
Sep 11 18:45:21.097909 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:21.097884    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb"
Sep 11 18:45:21.966357 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:21.966325    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb"
Sep 11 18:45:21.966541 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:21.966331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb" event={"ID":"8530cdd1-b251-4b14-93ad-46fe3695d58c","Type":"ContainerDied","Data":"69891518a9fada584ba50bd67e26ddd83d30119d7843d2a7a675261b681ebbd4"}
Sep 11 18:45:21.966541 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:21.966462    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69891518a9fada584ba50bd67e26ddd83d30119d7843d2a7a675261b681ebbd4"
Sep 11 18:45:33.808610 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:33.808536    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr"]
Sep 11 18:45:33.812788 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:33.812759    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-883e85-29819205-5ldqr"]
Sep 11 18:45:33.833308 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:33.833280    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"]
Sep 11 18:45:33.833555 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:33.833498    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9" podUID="197ccb8d-73cf-41d6-a18c-d263d7c53700" containerName="payload-processing" containerID="cri-o://bacd3335fad2fc98b6fb70771309abb7784fe2df602954b7c4007dbedda56dc9" gracePeriod=30
Sep 11 18:45:33.838913 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:33.838884    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"]
Sep 11 18:45:33.839326 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:33.839277    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm" podUID="d50e9b73-73fb-4700-a7a2-0c36000188ac" containerName="payload-pre-processing" containerID="cri-o://2ea0c0d109a999b8b012c8cf9e7b7de0b6a8a97019e4688a925cc17b077ad867" gracePeriod=30
Sep 11 18:45:33.938186 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:33.938151    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a5f3c0-6420-4606-9b39-ec06abb1cc48" path="/var/lib/kubelet/pods/52a5f3c0-6420-4606-9b39-ec06abb1cc48/volumes"
Sep 11 18:45:34.029823 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.029792    2578 generic.go:356] "Generic (PLEG): container finished" podID="d50e9b73-73fb-4700-a7a2-0c36000188ac" containerID="2ea0c0d109a999b8b012c8cf9e7b7de0b6a8a97019e4688a925cc17b077ad867" exitCode=0
Sep 11 18:45:34.029985 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.029862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm" event={"ID":"d50e9b73-73fb-4700-a7a2-0c36000188ac","Type":"ContainerDied","Data":"2ea0c0d109a999b8b012c8cf9e7b7de0b6a8a97019e4688a925cc17b077ad867"}
Sep 11 18:45:34.031501 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.031479    2578 generic.go:356] "Generic (PLEG): container finished" podID="197ccb8d-73cf-41d6-a18c-d263d7c53700" containerID="bacd3335fad2fc98b6fb70771309abb7784fe2df602954b7c4007dbedda56dc9" exitCode=0
Sep 11 18:45:34.031608 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.031511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9" event={"ID":"197ccb8d-73cf-41d6-a18c-d263d7c53700","Type":"ContainerDied","Data":"bacd3335fad2fc98b6fb70771309abb7784fe2df602954b7c4007dbedda56dc9"}
Sep 11 18:45:34.121350 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.121325    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:45:34.127997 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.127973    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:45:34.174231 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.174198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgkfv\" (UniqueName: \"kubernetes.io/projected/197ccb8d-73cf-41d6-a18c-d263d7c53700-kube-api-access-jgkfv\") pod \"197ccb8d-73cf-41d6-a18c-d263d7c53700\" (UID: \"197ccb8d-73cf-41d6-a18c-d263d7c53700\") "
Sep 11 18:45:34.174364 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.174316    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197ccb8d-73cf-41d6-a18c-d263d7c53700-plugins-config-volume\") pod \"197ccb8d-73cf-41d6-a18c-d263d7c53700\" (UID: \"197ccb8d-73cf-41d6-a18c-d263d7c53700\") "
Sep 11 18:45:34.174364 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.174360    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkz46\" (UniqueName: \"kubernetes.io/projected/d50e9b73-73fb-4700-a7a2-0c36000188ac-kube-api-access-pkz46\") pod \"d50e9b73-73fb-4700-a7a2-0c36000188ac\" (UID: \"d50e9b73-73fb-4700-a7a2-0c36000188ac\") "
Sep 11 18:45:34.174479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.174411    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50e9b73-73fb-4700-a7a2-0c36000188ac-plugins-config-volume\") pod \"d50e9b73-73fb-4700-a7a2-0c36000188ac\" (UID: \"d50e9b73-73fb-4700-a7a2-0c36000188ac\") "
Sep 11 18:45:34.174814 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.174735    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197ccb8d-73cf-41d6-a18c-d263d7c53700-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "197ccb8d-73cf-41d6-a18c-d263d7c53700" (UID: "197ccb8d-73cf-41d6-a18c-d263d7c53700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:45:34.174947 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.174821    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197ccb8d-73cf-41d6-a18c-d263d7c53700-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:45:34.174947 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.174900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50e9b73-73fb-4700-a7a2-0c36000188ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d50e9b73-73fb-4700-a7a2-0c36000188ac" (UID: "d50e9b73-73fb-4700-a7a2-0c36000188ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:45:34.176440 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.176408    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197ccb8d-73cf-41d6-a18c-d263d7c53700-kube-api-access-jgkfv" (OuterVolumeSpecName: "kube-api-access-jgkfv") pod "197ccb8d-73cf-41d6-a18c-d263d7c53700" (UID: "197ccb8d-73cf-41d6-a18c-d263d7c53700"). InnerVolumeSpecName "kube-api-access-jgkfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:45:34.176691 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.176669    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50e9b73-73fb-4700-a7a2-0c36000188ac-kube-api-access-pkz46" (OuterVolumeSpecName: "kube-api-access-pkz46") pod "d50e9b73-73fb-4700-a7a2-0c36000188ac" (UID: "d50e9b73-73fb-4700-a7a2-0c36000188ac"). InnerVolumeSpecName "kube-api-access-pkz46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:45:34.275623 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.275585    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkz46\" (UniqueName: \"kubernetes.io/projected/d50e9b73-73fb-4700-a7a2-0c36000188ac-kube-api-access-pkz46\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:45:34.275623 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.275615    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50e9b73-73fb-4700-a7a2-0c36000188ac-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:45:34.275623 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:34.275624    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgkfv\" (UniqueName: \"kubernetes.io/projected/197ccb8d-73cf-41d6-a18c-d263d7c53700-kube-api-access-jgkfv\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:45:35.038782 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.038752    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"
Sep 11 18:45:35.038782 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.038779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm" event={"ID":"d50e9b73-73fb-4700-a7a2-0c36000188ac","Type":"ContainerDied","Data":"7d8735114b6de8686f644fcdfe089762fb87dd7442e9592285f4132d3f5d89d9"}
Sep 11 18:45:35.039257 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.038840    2578 scope.go:117] "RemoveContainer" containerID="2ea0c0d109a999b8b012c8cf9e7b7de0b6a8a97019e4688a925cc17b077ad867"
Sep 11 18:45:35.040194 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.040177    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"
Sep 11 18:45:35.040272 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.040176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9" event={"ID":"197ccb8d-73cf-41d6-a18c-d263d7c53700","Type":"ContainerDied","Data":"6597d94209d2566c5ff62c81bff3ce5d9a37f80586f91e1e15bc542f32c1240b"}
Sep 11 18:45:35.048492 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.048468    2578 scope.go:117] "RemoveContainer" containerID="bacd3335fad2fc98b6fb70771309abb7784fe2df602954b7c4007dbedda56dc9"
Sep 11 18:45:35.064626 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.064603    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"]
Sep 11 18:45:35.068829 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.068809    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-883e85-594b9cb4c8-fq4pm"]
Sep 11 18:45:35.080034 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.080011    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"]
Sep 11 18:45:35.084374 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.084350    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-883e85-67df75b6b9-bgrh9"]
Sep 11 18:45:35.938678 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.938633    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="197ccb8d-73cf-41d6-a18c-d263d7c53700" path="/var/lib/kubelet/pods/197ccb8d-73cf-41d6-a18c-d263d7c53700/volumes"
Sep 11 18:45:35.939071 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:35.939054    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50e9b73-73fb-4700-a7a2-0c36000188ac" path="/var/lib/kubelet/pods/d50e9b73-73fb-4700-a7a2-0c36000188ac/volumes"
Sep 11 18:45:56.459533 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.459492    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh"]
Sep 11 18:45:56.462143 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.462120    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5cbc66-29819205-bhwxh"]
Sep 11 18:45:56.486421 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.486383    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"]
Sep 11 18:45:56.486675 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.486626    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb" podUID="979040e9-ca5c-4741-aebc-2bd65e1c7b60" containerName="payload-processing" containerID="cri-o://9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531" gracePeriod=30
Sep 11 18:45:56.743007 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.742982    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:45:56.801287 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.801252    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979040e9-ca5c-4741-aebc-2bd65e1c7b60-plugins-config-volume\") pod \"979040e9-ca5c-4741-aebc-2bd65e1c7b60\" (UID: \"979040e9-ca5c-4741-aebc-2bd65e1c7b60\") "
Sep 11 18:45:56.801473 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.801315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgtv5\" (UniqueName: \"kubernetes.io/projected/979040e9-ca5c-4741-aebc-2bd65e1c7b60-kube-api-access-pgtv5\") pod \"979040e9-ca5c-4741-aebc-2bd65e1c7b60\" (UID: \"979040e9-ca5c-4741-aebc-2bd65e1c7b60\") "
Sep 11 18:45:56.801735 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.801710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979040e9-ca5c-4741-aebc-2bd65e1c7b60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "979040e9-ca5c-4741-aebc-2bd65e1c7b60" (UID: "979040e9-ca5c-4741-aebc-2bd65e1c7b60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:45:56.803396 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.803373    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979040e9-ca5c-4741-aebc-2bd65e1c7b60-kube-api-access-pgtv5" (OuterVolumeSpecName: "kube-api-access-pgtv5") pod "979040e9-ca5c-4741-aebc-2bd65e1c7b60" (UID: "979040e9-ca5c-4741-aebc-2bd65e1c7b60"). InnerVolumeSpecName "kube-api-access-pgtv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:45:56.902309 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.902276    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgtv5\" (UniqueName: \"kubernetes.io/projected/979040e9-ca5c-4741-aebc-2bd65e1c7b60-kube-api-access-pgtv5\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:45:56.902309 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:56.902305    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979040e9-ca5c-4741-aebc-2bd65e1c7b60-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:45:57.153363 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.153326    2578 generic.go:356] "Generic (PLEG): container finished" podID="979040e9-ca5c-4741-aebc-2bd65e1c7b60" containerID="9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531" exitCode=0
Sep 11 18:45:57.153543 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.153375    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"
Sep 11 18:45:57.153543 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.153402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb" event={"ID":"979040e9-ca5c-4741-aebc-2bd65e1c7b60","Type":"ContainerDied","Data":"9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531"}
Sep 11 18:45:57.153543 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.153471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb" event={"ID":"979040e9-ca5c-4741-aebc-2bd65e1c7b60","Type":"ContainerDied","Data":"aa32c6cec2d3a52adf95206d6fe173c37ce1f96907231dafa89753a947cd4af4"}
Sep 11 18:45:57.153543 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.153488    2578 scope.go:117] "RemoveContainer" containerID="9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531"
Sep 11 18:45:57.162927 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.162906    2578 scope.go:117] "RemoveContainer" containerID="9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531"
Sep 11 18:45:57.163203 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:45:57.163182    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531\": container with ID starting with 9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531 not found: ID does not exist" containerID="9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531"
Sep 11 18:45:57.163252 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.163214    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531"} err="failed to get container status \"9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531\": rpc error: code = NotFound desc = could not find container \"9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531\": container with ID starting with 9c17bff59102812b09f7e72904a2832141268e50bc6a03d08fac799515be6531 not found: ID does not exist"
Sep 11 18:45:57.175705 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.175681    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"]
Sep 11 18:45:57.181789 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.181768    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5cbc66-65879468d7-968wb"]
Sep 11 18:45:57.938982 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.938938    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="979040e9-ca5c-4741-aebc-2bd65e1c7b60" path="/var/lib/kubelet/pods/979040e9-ca5c-4741-aebc-2bd65e1c7b60/volumes"
Sep 11 18:45:57.939450 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:45:57.939364    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd4874b1-2d71-4b9a-b6e6-bbea4ea9d31e" path="/var/lib/kubelet/pods/bd4874b1-2d71-4b9a-b6e6-bbea4ea9d31e/volumes"
Sep 11 18:46:01.587196 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:01.587159    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"]
Sep 11 18:46:01.592457 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:01.592428    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"]
Sep 11 18:46:01.600558 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:01.600526    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"]
Sep 11 18:46:01.609876 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:01.609843    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"]
Sep 11 18:46:14.068063 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.068025    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"]
Sep 11 18:46:14.070446 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.068257    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq" podUID="2b2fa513-81f4-46f1-aec8-e1aa167ef874" containerName="payload-processing" containerID="cri-o://ac1414881c8388c4eb383d6c7f481e47c8bd7e691be375ad0e71a50fc9a41f21" gracePeriod=30
Sep 11 18:46:14.209516 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.209482    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb"]
Sep 11 18:46:14.212483 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.212459    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72c98d-29819205-dp6qb"]
Sep 11 18:46:14.242102 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.242062    2578 generic.go:356] "Generic (PLEG): container finished" podID="2b2fa513-81f4-46f1-aec8-e1aa167ef874" containerID="ac1414881c8388c4eb383d6c7f481e47c8bd7e691be375ad0e71a50fc9a41f21" exitCode=0
Sep 11 18:46:14.242303 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.242138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq" event={"ID":"2b2fa513-81f4-46f1-aec8-e1aa167ef874","Type":"ContainerDied","Data":"ac1414881c8388c4eb383d6c7f481e47c8bd7e691be375ad0e71a50fc9a41f21"}
Sep 11 18:46:14.340423 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.340380    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:46:14.464839 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.464806    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2fa513-81f4-46f1-aec8-e1aa167ef874-plugins-config-volume\") pod \"2b2fa513-81f4-46f1-aec8-e1aa167ef874\" (UID: \"2b2fa513-81f4-46f1-aec8-e1aa167ef874\") "
Sep 11 18:46:14.465021 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.464889    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjdvd\" (UniqueName: \"kubernetes.io/projected/2b2fa513-81f4-46f1-aec8-e1aa167ef874-kube-api-access-jjdvd\") pod \"2b2fa513-81f4-46f1-aec8-e1aa167ef874\" (UID: \"2b2fa513-81f4-46f1-aec8-e1aa167ef874\") "
Sep 11 18:46:14.465207 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.465177    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2fa513-81f4-46f1-aec8-e1aa167ef874-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b2fa513-81f4-46f1-aec8-e1aa167ef874" (UID: "2b2fa513-81f4-46f1-aec8-e1aa167ef874"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:46:14.465342 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.465318    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2fa513-81f4-46f1-aec8-e1aa167ef874-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:14.466935 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.466913    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2fa513-81f4-46f1-aec8-e1aa167ef874-kube-api-access-jjdvd" (OuterVolumeSpecName: "kube-api-access-jjdvd") pod "2b2fa513-81f4-46f1-aec8-e1aa167ef874" (UID: "2b2fa513-81f4-46f1-aec8-e1aa167ef874"). InnerVolumeSpecName "kube-api-access-jjdvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:46:14.567033 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:14.566986    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjdvd\" (UniqueName: \"kubernetes.io/projected/2b2fa513-81f4-46f1-aec8-e1aa167ef874-kube-api-access-jjdvd\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:15.248567 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:15.248533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq" event={"ID":"2b2fa513-81f4-46f1-aec8-e1aa167ef874","Type":"ContainerDied","Data":"1dd454feb3a97d747c8091383be9402f940ba6effe621abb03b773015bc6943e"}
Sep 11 18:46:15.248567 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:15.248562    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"
Sep 11 18:46:15.249083 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:15.248594    2578 scope.go:117] "RemoveContainer" containerID="ac1414881c8388c4eb383d6c7f481e47c8bd7e691be375ad0e71a50fc9a41f21"
Sep 11 18:46:15.271850 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:15.271792    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"]
Sep 11 18:46:15.278994 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:15.278965    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-200c07-647469c66c-dvcfq"]
Sep 11 18:46:15.938037 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:15.937999    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2fa513-81f4-46f1-aec8-e1aa167ef874" path="/var/lib/kubelet/pods/2b2fa513-81f4-46f1-aec8-e1aa167ef874/volumes"
Sep 11 18:46:15.938355 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:15.938341    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8530cdd1-b251-4b14-93ad-46fe3695d58c" path="/var/lib/kubelet/pods/8530cdd1-b251-4b14-93ad-46fe3695d58c/volumes"
Sep 11 18:46:16.607054 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.607015    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr" podUID="22cce2f0-187d-4593-be87-c8441b9dc9b6" containerName="main" containerID="cri-o://bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02" gracePeriod=45
Sep 11 18:46:16.611364 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.611211    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855" podUID="f1261164-4a5b-48df-b1e4-8563bde8a2be" containerName="main" containerID="cri-o://f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532" gracePeriod=45
Sep 11 18:46:16.623068 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.623037    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx" podUID="b12ecfcc-4797-4516-b42c-175e83500be0" containerName="main" containerID="cri-o://160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e" gracePeriod=45
Sep 11 18:46:16.628418 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.628370    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk" podUID="d8c5cb61-104b-438a-858c-0b3a51eac6dc" containerName="main" containerID="cri-o://c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b" gracePeriod=45
Sep 11 18:46:16.881297 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.881265    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:46:16.916236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.916207    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:46:16.957670 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.957629    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:46:16.960972 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.960946    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:46:16.991099 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1261164-4a5b-48df-b1e4-8563bde8a2be-tls-certs\") pod \"f1261164-4a5b-48df-b1e4-8563bde8a2be\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") "
Sep 11 18:46:16.991269 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991117    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-tmp-dir\") pod \"f1261164-4a5b-48df-b1e4-8563bde8a2be\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") "
Sep 11 18:46:16.991269 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991135    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-dshm\") pod \"b12ecfcc-4797-4516-b42c-175e83500be0\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") "
Sep 11 18:46:16.991269 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991188    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-model-cache\") pod \"f1261164-4a5b-48df-b1e4-8563bde8a2be\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") "
Sep 11 18:46:16.991269 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-tmp-dir\") pod \"b12ecfcc-4797-4516-b42c-175e83500be0\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") "
Sep 11 18:46:16.991528 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-model-cache\") pod \"b12ecfcc-4797-4516-b42c-175e83500be0\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") "
Sep 11 18:46:16.991528 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991297    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj6kx\" (UniqueName: \"kubernetes.io/projected/f1261164-4a5b-48df-b1e4-8563bde8a2be-kube-api-access-rj6kx\") pod \"f1261164-4a5b-48df-b1e4-8563bde8a2be\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") "
Sep 11 18:46:16.991528 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-home\") pod \"b12ecfcc-4797-4516-b42c-175e83500be0\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") "
Sep 11 18:46:16.991528 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991361    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12ecfcc-4797-4516-b42c-175e83500be0-tls-certs\") pod \"b12ecfcc-4797-4516-b42c-175e83500be0\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") "
Sep 11 18:46:16.991528 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1261164-4a5b-48df-b1e4-8563bde8a2be" (UID: "f1261164-4a5b-48df-b1e4-8563bde8a2be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:16.991528 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991412    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-dshm\") pod \"f1261164-4a5b-48df-b1e4-8563bde8a2be\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") "
Sep 11 18:46:16.991528 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991418    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1261164-4a5b-48df-b1e4-8563bde8a2be" (UID: "f1261164-4a5b-48df-b1e4-8563bde8a2be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:16.991528 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991455    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-home\") pod \"f1261164-4a5b-48df-b1e4-8563bde8a2be\" (UID: \"f1261164-4a5b-48df-b1e4-8563bde8a2be\") "
Sep 11 18:46:16.991528 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991490    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8dtk\" (UniqueName: \"kubernetes.io/projected/b12ecfcc-4797-4516-b42c-175e83500be0-kube-api-access-m8dtk\") pod \"b12ecfcc-4797-4516-b42c-175e83500be0\" (UID: \"b12ecfcc-4797-4516-b42c-175e83500be0\") "
Sep 11 18:46:16.992080 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991614    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b12ecfcc-4797-4516-b42c-175e83500be0" (UID: "b12ecfcc-4797-4516-b42c-175e83500be0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:16.992080 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.991688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-model-cache" (OuterVolumeSpecName: "model-cache") pod "b12ecfcc-4797-4516-b42c-175e83500be0" (UID: "b12ecfcc-4797-4516-b42c-175e83500be0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:16.992593 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.992567    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-home" (OuterVolumeSpecName: "home") pod "b12ecfcc-4797-4516-b42c-175e83500be0" (UID: "b12ecfcc-4797-4516-b42c-175e83500be0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:16.992593 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.992574    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-home" (OuterVolumeSpecName: "home") pod "f1261164-4a5b-48df-b1e4-8563bde8a2be" (UID: "f1261164-4a5b-48df-b1e4-8563bde8a2be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:16.992759 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.992735    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:16.992821 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.992781    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:16.992821 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.992797    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:16.992821 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.992809    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:16.992969 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.992821    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:16.992969 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.992834    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:16.995472 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.995450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12ecfcc-4797-4516-b42c-175e83500be0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b12ecfcc-4797-4516-b42c-175e83500be0" (UID: "b12ecfcc-4797-4516-b42c-175e83500be0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:46:16.995603 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.995483    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12ecfcc-4797-4516-b42c-175e83500be0-kube-api-access-m8dtk" (OuterVolumeSpecName: "kube-api-access-m8dtk") pod "b12ecfcc-4797-4516-b42c-175e83500be0" (UID: "b12ecfcc-4797-4516-b42c-175e83500be0"). InnerVolumeSpecName "kube-api-access-m8dtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:46:16.995770 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.995750    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-dshm" (OuterVolumeSpecName: "dshm") pod "f1261164-4a5b-48df-b1e4-8563bde8a2be" (UID: "f1261164-4a5b-48df-b1e4-8563bde8a2be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:16.995861 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.995843    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1261164-4a5b-48df-b1e4-8563bde8a2be-kube-api-access-rj6kx" (OuterVolumeSpecName: "kube-api-access-rj6kx") pod "f1261164-4a5b-48df-b1e4-8563bde8a2be" (UID: "f1261164-4a5b-48df-b1e4-8563bde8a2be"). InnerVolumeSpecName "kube-api-access-rj6kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:46:16.996159 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.996133    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-dshm" (OuterVolumeSpecName: "dshm") pod "b12ecfcc-4797-4516-b42c-175e83500be0" (UID: "b12ecfcc-4797-4516-b42c-175e83500be0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:16.996479 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:16.996458    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1261164-4a5b-48df-b1e4-8563bde8a2be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1261164-4a5b-48df-b1e4-8563bde8a2be" (UID: "f1261164-4a5b-48df-b1e4-8563bde8a2be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:46:17.094002 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.093965    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-tmp-dir\") pod \"22cce2f0-187d-4593-be87-c8441b9dc9b6\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") "
Sep 11 18:46:17.094177 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094011    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-home\") pod \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") "
Sep 11 18:46:17.094177 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tls-certs\") pod \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") "
Sep 11 18:46:17.094177 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cce2f0-187d-4593-be87-c8441b9dc9b6-tls-certs\") pod \"22cce2f0-187d-4593-be87-c8441b9dc9b6\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") "
Sep 11 18:46:17.094177 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094087    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-dshm\") pod \"22cce2f0-187d-4593-be87-c8441b9dc9b6\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") "
Sep 11 18:46:17.094177 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094111    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tmp-dir\") pod \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") "
Sep 11 18:46:17.094464 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094184    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-model-cache\") pod \"22cce2f0-187d-4593-be87-c8441b9dc9b6\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") "
Sep 11 18:46:17.094464 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094209    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5468z\" (UniqueName: \"kubernetes.io/projected/22cce2f0-187d-4593-be87-c8441b9dc9b6-kube-api-access-5468z\") pod \"22cce2f0-187d-4593-be87-c8441b9dc9b6\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") "
Sep 11 18:46:17.094464 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094235    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-home\") pod \"22cce2f0-187d-4593-be87-c8441b9dc9b6\" (UID: \"22cce2f0-187d-4593-be87-c8441b9dc9b6\") "
Sep 11 18:46:17.094464 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094252    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22cce2f0-187d-4593-be87-c8441b9dc9b6" (UID: "22cce2f0-187d-4593-be87-c8441b9dc9b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:17.094464 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094279    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-home" (OuterVolumeSpecName: "home") pod "d8c5cb61-104b-438a-858c-0b3a51eac6dc" (UID: "d8c5cb61-104b-438a-858c-0b3a51eac6dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:17.094464 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094327    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6grn\" (UniqueName: \"kubernetes.io/projected/d8c5cb61-104b-438a-858c-0b3a51eac6dc-kube-api-access-l6grn\") pod \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") "
Sep 11 18:46:17.094464 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094360    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-dshm\") pod \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") "
Sep 11 18:46:17.094464 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-model-cache\") pod \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\" (UID: \"d8c5cb61-104b-438a-858c-0b3a51eac6dc\") "
Sep 11 18:46:17.094916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094537    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8c5cb61-104b-438a-858c-0b3a51eac6dc" (UID: "d8c5cb61-104b-438a-858c-0b3a51eac6dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:17.094916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094635    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8c5cb61-104b-438a-858c-0b3a51eac6dc" (UID: "d8c5cb61-104b-438a-858c-0b3a51eac6dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:17.094916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "22cce2f0-187d-4593-be87-c8441b9dc9b6" (UID: "22cce2f0-187d-4593-be87-c8441b9dc9b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:17.095080 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094966    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-home" (OuterVolumeSpecName: "home") pod "22cce2f0-187d-4593-be87-c8441b9dc9b6" (UID: "22cce2f0-187d-4593-be87-c8441b9dc9b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:17.095080 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.094994    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj6kx\" (UniqueName: \"kubernetes.io/projected/f1261164-4a5b-48df-b1e4-8563bde8a2be-kube-api-access-rj6kx\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.095080 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.095019    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12ecfcc-4797-4516-b42c-175e83500be0-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.095080 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.095035    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.095080 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.095048    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1261164-4a5b-48df-b1e4-8563bde8a2be-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.095080 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.095060    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.095080 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.095073    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.095422 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.095086    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8dtk\" (UniqueName: \"kubernetes.io/projected/b12ecfcc-4797-4516-b42c-175e83500be0-kube-api-access-m8dtk\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.095422 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.095101    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.095422 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.095115    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1261164-4a5b-48df-b1e4-8563bde8a2be-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.095422 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.095127    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12ecfcc-4797-4516-b42c-175e83500be0-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.095422 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.095140    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.096414 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.096390    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8c5cb61-104b-438a-858c-0b3a51eac6dc" (UID: "d8c5cb61-104b-438a-858c-0b3a51eac6dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:46:17.096504 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.096440    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-dshm" (OuterVolumeSpecName: "dshm") pod "22cce2f0-187d-4593-be87-c8441b9dc9b6" (UID: "22cce2f0-187d-4593-be87-c8441b9dc9b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:17.096670 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.096629    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cce2f0-187d-4593-be87-c8441b9dc9b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22cce2f0-187d-4593-be87-c8441b9dc9b6" (UID: "22cce2f0-187d-4593-be87-c8441b9dc9b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:46:17.096857 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.096832    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-dshm" (OuterVolumeSpecName: "dshm") pod "d8c5cb61-104b-438a-858c-0b3a51eac6dc" (UID: "d8c5cb61-104b-438a-858c-0b3a51eac6dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:17.096945 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.096892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cce2f0-187d-4593-be87-c8441b9dc9b6-kube-api-access-5468z" (OuterVolumeSpecName: "kube-api-access-5468z") pod "22cce2f0-187d-4593-be87-c8441b9dc9b6" (UID: "22cce2f0-187d-4593-be87-c8441b9dc9b6"). InnerVolumeSpecName "kube-api-access-5468z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:46:17.096988 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.096967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c5cb61-104b-438a-858c-0b3a51eac6dc-kube-api-access-l6grn" (OuterVolumeSpecName: "kube-api-access-l6grn") pod "d8c5cb61-104b-438a-858c-0b3a51eac6dc" (UID: "d8c5cb61-104b-438a-858c-0b3a51eac6dc"). InnerVolumeSpecName "kube-api-access-l6grn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:46:17.196687 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.196582    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cce2f0-187d-4593-be87-c8441b9dc9b6-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.196687 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.196611    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.196687 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.196620    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5468z\" (UniqueName: \"kubernetes.io/projected/22cce2f0-187d-4593-be87-c8441b9dc9b6-kube-api-access-5468z\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.196687 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.196630    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cce2f0-187d-4593-be87-c8441b9dc9b6-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.196687 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.196639    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6grn\" (UniqueName: \"kubernetes.io/projected/d8c5cb61-104b-438a-858c-0b3a51eac6dc-kube-api-access-l6grn\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.196687 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.196673    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c5cb61-104b-438a-858c-0b3a51eac6dc-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.196687 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.196686    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c5cb61-104b-438a-858c-0b3a51eac6dc-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:17.260030 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.259992    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1261164-4a5b-48df-b1e4-8563bde8a2be" containerID="f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532" exitCode=0
Sep 11 18:46:17.260181 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.260039    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"
Sep 11 18:46:17.260181 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.260078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855" event={"ID":"f1261164-4a5b-48df-b1e4-8563bde8a2be","Type":"ContainerDied","Data":"f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532"}
Sep 11 18:46:17.260181 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.260119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855" event={"ID":"f1261164-4a5b-48df-b1e4-8563bde8a2be","Type":"ContainerDied","Data":"f173922e6f48ad7a2e0bca0fa3bd652703992d64aa13ac5c76cbe2803d57a825"}
Sep 11 18:46:17.260181 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.260138    2578 scope.go:117] "RemoveContainer" containerID="f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532"
Sep 11 18:46:17.261774 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.261452    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8c5cb61-104b-438a-858c-0b3a51eac6dc" containerID="c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b" exitCode=0
Sep 11 18:46:17.261774 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.261481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk" event={"ID":"d8c5cb61-104b-438a-858c-0b3a51eac6dc","Type":"ContainerDied","Data":"c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b"}
Sep 11 18:46:17.261774 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.261504    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"
Sep 11 18:46:17.261774 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.261516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk" event={"ID":"d8c5cb61-104b-438a-858c-0b3a51eac6dc","Type":"ContainerDied","Data":"1ca014899a46352135eadd26054f9e4fb6968e42319bb0f18872eafc3e221b4c"}
Sep 11 18:46:17.263268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.263248    2578 generic.go:356] "Generic (PLEG): container finished" podID="22cce2f0-187d-4593-be87-c8441b9dc9b6" containerID="bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02" exitCode=0
Sep 11 18:46:17.263388 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.263290    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"
Sep 11 18:46:17.263388 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.263318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr" event={"ID":"22cce2f0-187d-4593-be87-c8441b9dc9b6","Type":"ContainerDied","Data":"bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02"}
Sep 11 18:46:17.263388 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.263342    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr" event={"ID":"22cce2f0-187d-4593-be87-c8441b9dc9b6","Type":"ContainerDied","Data":"21ab7d20cb62e18995e82d2cc54f94e43864d0d105e144ecf73c8eda8d7fd4e8"}
Sep 11 18:46:17.264841 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.264822    2578 generic.go:356] "Generic (PLEG): container finished" podID="b12ecfcc-4797-4516-b42c-175e83500be0" containerID="160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e" exitCode=0
Sep 11 18:46:17.264952 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.264897    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx" event={"ID":"b12ecfcc-4797-4516-b42c-175e83500be0","Type":"ContainerDied","Data":"160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e"}
Sep 11 18:46:17.264952 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.264920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx" event={"ID":"b12ecfcc-4797-4516-b42c-175e83500be0","Type":"ContainerDied","Data":"2b375e32aaa7b5210099559238731f685a093f4dc3762d3d5c4aad377d8b2315"}
Sep 11 18:46:17.264952 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.264933    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"
Sep 11 18:46:17.271365 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.271349    2578 scope.go:117] "RemoveContainer" containerID="f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532"
Sep 11 18:46:17.271662 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:46:17.271631    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532\": container with ID starting with f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532 not found: ID does not exist" containerID="f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532"
Sep 11 18:46:17.271743 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.271676    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532"} err="failed to get container status \"f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532\": rpc error: code = NotFound desc = could not find container \"f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532\": container with ID starting with f963c88735cc5cf11c7fb87a17dcff7e2bee56c513f3b2677065b8a55144e532 not found: ID does not exist"
Sep 11 18:46:17.271743 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.271697    2578 scope.go:117] "RemoveContainer" containerID="c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b"
Sep 11 18:46:17.281240 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.281224    2578 scope.go:117] "RemoveContainer" containerID="c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b"
Sep 11 18:46:17.281484 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:46:17.281467    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b\": container with ID starting with c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b not found: ID does not exist" containerID="c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b"
Sep 11 18:46:17.281544 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.281490    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b"} err="failed to get container status \"c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b\": rpc error: code = NotFound desc = could not find container \"c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b\": container with ID starting with c746b9ad9cda966a028366f789b05b469e83b5716eb63b6b16574eb58548de0b not found: ID does not exist"
Sep 11 18:46:17.281544 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.281508    2578 scope.go:117] "RemoveContainer" containerID="bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02"
Sep 11 18:46:17.290171 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.290154    2578 scope.go:117] "RemoveContainer" containerID="bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02"
Sep 11 18:46:17.290470 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.290448    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"]
Sep 11 18:46:17.290578 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:46:17.290516    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02\": container with ID starting with bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02 not found: ID does not exist" containerID="bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02"
Sep 11 18:46:17.290578 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.290542    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02"} err="failed to get container status \"bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02\": rpc error: code = NotFound desc = could not find container \"bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02\": container with ID starting with bcd464aa8abe4574a93ecc60df78c3b1969e08a78a8ae74a5fee4e9203bccf02 not found: ID does not exist"
Sep 11 18:46:17.290578 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.290563    2578 scope.go:117] "RemoveContainer" containerID="160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e"
Sep 11 18:46:17.293168 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.293146    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-w4-200c07-kserve-784d7f8c57-b4855"]
Sep 11 18:46:17.298447 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.298421    2578 scope.go:117] "RemoveContainer" containerID="160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e"
Sep 11 18:46:17.298729 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:46:17.298710    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e\": container with ID starting with 160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e not found: ID does not exist" containerID="160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e"
Sep 11 18:46:17.298789 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.298735    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e"} err="failed to get container status \"160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e\": rpc error: code = NotFound desc = could not find container \"160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e\": container with ID starting with 160ee09b44ae2391a2cb332b569cb3b02a271b2f24108e1df0d73ece0fbc5c6e not found: ID does not exist"
Sep 11 18:46:17.303529 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.303509    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"]
Sep 11 18:46:17.306910 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.306883    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/premium-w4-200c07-kserve-768cf64954-mm4tx"]
Sep 11 18:46:17.317491 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.317460    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"]
Sep 11 18:46:17.323504 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.323472    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/distinct-2-w4-200c07-kserve-66cf58b5fd-xmkkr"]
Sep 11 18:46:17.335955 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.335930    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"]
Sep 11 18:46:17.340036 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.340017    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-200c07/simulator-w4-200c07-kserve-56847d8f59-fj2lk"]
Sep 11 18:46:17.937948 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.937873    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22cce2f0-187d-4593-be87-c8441b9dc9b6" path="/var/lib/kubelet/pods/22cce2f0-187d-4593-be87-c8441b9dc9b6/volumes"
Sep 11 18:46:17.938312 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.938296    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12ecfcc-4797-4516-b42c-175e83500be0" path="/var/lib/kubelet/pods/b12ecfcc-4797-4516-b42c-175e83500be0/volumes"
Sep 11 18:46:17.938688 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.938675    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c5cb61-104b-438a-858c-0b3a51eac6dc" path="/var/lib/kubelet/pods/d8c5cb61-104b-438a-858c-0b3a51eac6dc/volumes"
Sep 11 18:46:17.939022 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:17.939011    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1261164-4a5b-48df-b1e4-8563bde8a2be" path="/var/lib/kubelet/pods/f1261164-4a5b-48df-b1e4-8563bde8a2be/volumes"
Sep 11 18:46:32.684492 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:32.684453    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"]
Sep 11 18:46:32.684981 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:32.684733    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn" podUID="646aa882-298f-4808-ac03-ac41a18ce9b9" containerName="payload-pre-processing" containerID="cri-o://24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b" gracePeriod=30
Sep 11 18:46:32.939337 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:32.939262    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:46:33.049977 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.049938    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g52h\" (UniqueName: \"kubernetes.io/projected/646aa882-298f-4808-ac03-ac41a18ce9b9-kube-api-access-9g52h\") pod \"646aa882-298f-4808-ac03-ac41a18ce9b9\" (UID: \"646aa882-298f-4808-ac03-ac41a18ce9b9\") "
Sep 11 18:46:33.050154 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.050067    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646aa882-298f-4808-ac03-ac41a18ce9b9-plugins-config-volume\") pod \"646aa882-298f-4808-ac03-ac41a18ce9b9\" (UID: \"646aa882-298f-4808-ac03-ac41a18ce9b9\") "
Sep 11 18:46:33.050466 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.050435    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646aa882-298f-4808-ac03-ac41a18ce9b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "646aa882-298f-4808-ac03-ac41a18ce9b9" (UID: "646aa882-298f-4808-ac03-ac41a18ce9b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:46:33.052188 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.052159    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646aa882-298f-4808-ac03-ac41a18ce9b9-kube-api-access-9g52h" (OuterVolumeSpecName: "kube-api-access-9g52h") pod "646aa882-298f-4808-ac03-ac41a18ce9b9" (UID: "646aa882-298f-4808-ac03-ac41a18ce9b9"). InnerVolumeSpecName "kube-api-access-9g52h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:46:33.151365 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.151334    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g52h\" (UniqueName: \"kubernetes.io/projected/646aa882-298f-4808-ac03-ac41a18ce9b9-kube-api-access-9g52h\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:33.151365 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.151360    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646aa882-298f-4808-ac03-ac41a18ce9b9-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:46:33.345589 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.345556    2578 generic.go:356] "Generic (PLEG): container finished" podID="646aa882-298f-4808-ac03-ac41a18ce9b9" containerID="24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b" exitCode=0
Sep 11 18:46:33.345756 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.345599    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"
Sep 11 18:46:33.345756 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.345622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn" event={"ID":"646aa882-298f-4808-ac03-ac41a18ce9b9","Type":"ContainerDied","Data":"24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b"}
Sep 11 18:46:33.345756 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.345689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn" event={"ID":"646aa882-298f-4808-ac03-ac41a18ce9b9","Type":"ContainerDied","Data":"d3033e87ddfbefbc21b2040d62c5643b50ac366a46ef905c4e19954473472874"}
Sep 11 18:46:33.345756 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.345712    2578 scope.go:117] "RemoveContainer" containerID="24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b"
Sep 11 18:46:33.354543 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.354526    2578 scope.go:117] "RemoveContainer" containerID="24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b"
Sep 11 18:46:33.354816 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:46:33.354798    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b\": container with ID starting with 24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b not found: ID does not exist" containerID="24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b"
Sep 11 18:46:33.354878 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.354825    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b"} err="failed to get container status \"24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b\": rpc error: code = NotFound desc = could not find container \"24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b\": container with ID starting with 24e86a64d86782d6022fc918f6446ce1a823656d3e2a3e9855658c2879522a5b not found: ID does not exist"
Sep 11 18:46:33.368999 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.368972    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"]
Sep 11 18:46:33.375176 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.375151    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-238bae-b7ff5df54-pbczn"]
Sep 11 18:46:33.937052 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:46:33.937016    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646aa882-298f-4808-ac03-ac41a18ce9b9" path="/var/lib/kubelet/pods/646aa882-298f-4808-ac03-ac41a18ce9b9/volumes"
Sep 11 18:47:00.948821 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.948727    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"]
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949125    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12ecfcc-4797-4516-b42c-175e83500be0" containerName="main"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949140    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12ecfcc-4797-4516-b42c-175e83500be0" containerName="main"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949154    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c5cb61-104b-438a-858c-0b3a51eac6dc" containerName="main"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949160    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c5cb61-104b-438a-858c-0b3a51eac6dc" containerName="main"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949168    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="197ccb8d-73cf-41d6-a18c-d263d7c53700" containerName="payload-processing"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949174    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="197ccb8d-73cf-41d6-a18c-d263d7c53700" containerName="payload-processing"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949182    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1261164-4a5b-48df-b1e4-8563bde8a2be" containerName="main"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949187    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1261164-4a5b-48df-b1e4-8563bde8a2be" containerName="main"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949194    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22cce2f0-187d-4593-be87-c8441b9dc9b6" containerName="main"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949200    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22cce2f0-187d-4593-be87-c8441b9dc9b6" containerName="main"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949210    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8530cdd1-b251-4b14-93ad-46fe3695d58c" containerName="cleanup"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949217    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8530cdd1-b251-4b14-93ad-46fe3695d58c" containerName="cleanup"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949225    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50e9b73-73fb-4700-a7a2-0c36000188ac" containerName="payload-pre-processing"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949230    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50e9b73-73fb-4700-a7a2-0c36000188ac" containerName="payload-pre-processing"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949238    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4874b1-2d71-4b9a-b6e6-bbea4ea9d31e" containerName="cleanup"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949243    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4874b1-2d71-4b9a-b6e6-bbea4ea9d31e" containerName="cleanup"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949250    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a5f3c0-6420-4606-9b39-ec06abb1cc48" containerName="cleanup"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949255    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a5f3c0-6420-4606-9b39-ec06abb1cc48" containerName="cleanup"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949266    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2fa513-81f4-46f1-aec8-e1aa167ef874" containerName="payload-processing"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949271    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2fa513-81f4-46f1-aec8-e1aa167ef874" containerName="payload-processing"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949276    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646aa882-298f-4808-ac03-ac41a18ce9b9" containerName="payload-pre-processing"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949282    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="646aa882-298f-4808-ac03-ac41a18ce9b9" containerName="payload-pre-processing"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949294    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="979040e9-ca5c-4741-aebc-2bd65e1c7b60" containerName="payload-processing"
Sep 11 18:47:00.949319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949301    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="979040e9-ca5c-4741-aebc-2bd65e1c7b60" containerName="payload-processing"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949350    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50e9b73-73fb-4700-a7a2-0c36000188ac" containerName="payload-pre-processing"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949359    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a5f3c0-6420-4606-9b39-ec06abb1cc48" containerName="cleanup"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949368    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4874b1-2d71-4b9a-b6e6-bbea4ea9d31e" containerName="cleanup"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949377    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22cce2f0-187d-4593-be87-c8441b9dc9b6" containerName="main"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949382    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2fa513-81f4-46f1-aec8-e1aa167ef874" containerName="payload-processing"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949386    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c5cb61-104b-438a-858c-0b3a51eac6dc" containerName="main"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949391    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="646aa882-298f-4808-ac03-ac41a18ce9b9" containerName="payload-pre-processing"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949396    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="979040e9-ca5c-4741-aebc-2bd65e1c7b60" containerName="payload-processing"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949410    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8530cdd1-b251-4b14-93ad-46fe3695d58c" containerName="cleanup"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949414    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12ecfcc-4797-4516-b42c-175e83500be0" containerName="main"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949420    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1261164-4a5b-48df-b1e4-8563bde8a2be" containerName="main"
Sep 11 18:47:00.950361 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.949425    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="197ccb8d-73cf-41d6-a18c-d263d7c53700" containerName="payload-processing"
Sep 11 18:47:00.958431 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.958391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:00.960393 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.960364    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"]
Sep 11 18:47:00.962997 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.962974    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90e71\"/\"openshift-service-ca.crt\""
Sep 11 18:47:00.963123 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.962973    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90e71\"/\"kube-root-ca.crt\""
Sep 11 18:47:00.963123 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.963070    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90e71\"/\"sub-test-model-d90e71-kserve-self-signed-certs\""
Sep 11 18:47:00.963249 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:00.963170    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90e71\"/\"default-dockercfg-p9zhk\""
Sep 11 18:47:01.115588 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.115548    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tmp-dir\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.115588 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.115588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-dshm\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.115811 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.115696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55hz\" (UniqueName: \"kubernetes.io/projected/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-kube-api-access-r55hz\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.115811 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.115743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tls-certs\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.115811 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.115780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-model-cache\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.115811 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.115812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-home\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.217076 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.216995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tmp-dir\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.217076 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.217037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-dshm\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.217257 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.217093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r55hz\" (UniqueName: \"kubernetes.io/projected/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-kube-api-access-r55hz\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.217257 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.217133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tls-certs\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.217257 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.217196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-model-cache\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.217257 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.217249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-home\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.217459 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.217439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tmp-dir\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.217556 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.217536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-home\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.217621 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.217600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-model-cache\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.219231 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.219216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-dshm\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.219571 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.219555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tls-certs\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.225941 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.225922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55hz\" (UniqueName: \"kubernetes.io/projected/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-kube-api-access-r55hz\") pod \"sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.269532 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.269507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:01.543860 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:01.543826    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"]
Sep 11 18:47:01.547299 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:47:01.547266    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddbfd5a1_32c3_47ac_9fdf_9fa2998575e4.slice/crio-690d1d57293bb64e51d50d43e530e749433a0325920348e49ebc23f83f1a5408 WatchSource:0}: Error finding container 690d1d57293bb64e51d50d43e530e749433a0325920348e49ebc23f83f1a5408: Status 404 returned error can't find the container with id 690d1d57293bb64e51d50d43e530e749433a0325920348e49ebc23f83f1a5408
Sep 11 18:47:02.485221 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:02.485180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl" event={"ID":"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4","Type":"ContainerStarted","Data":"a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83"}
Sep 11 18:47:02.485582 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:02.485232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl" event={"ID":"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4","Type":"ContainerStarted","Data":"690d1d57293bb64e51d50d43e530e749433a0325920348e49ebc23f83f1a5408"}
Sep 11 18:47:02.502672 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:02.502603    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl" podStartSLOduration=2.502589435 podStartE2EDuration="2.502589435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:47:02.500996321 +0000 UTC m=+1887.080260893" watchObservedRunningTime="2026-09-11 18:47:02.502589435 +0000 UTC m=+1887.081853995"
Sep 11 18:47:11.269743 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:11.269699    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:11.270193 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:11.269756    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:11.282327 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:11.282303    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:47:11.540596 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:47:11.540558    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:48:26.071847 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:26.071803    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"]
Sep 11 18:48:35.248382 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.248302    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"]
Sep 11 18:48:35.251754 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.251736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.254238 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.254217    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d90e71\"/\"rate-test-model-d90e71-kserve-self-signed-certs\""
Sep 11 18:48:35.259201 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.259180    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"]
Sep 11 18:48:35.376376 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.376337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-home\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.376376 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.376377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd2bacfe-d11d-431f-bdef-e69bf896271c-tls-certs\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.376624 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.376392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-tmp-dir\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.376624 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.376457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-model-cache\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.376624 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.376514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzpfv\" (UniqueName: \"kubernetes.io/projected/fd2bacfe-d11d-431f-bdef-e69bf896271c-kube-api-access-qzpfv\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.376822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.376677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-dshm\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.478094 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.478053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-home\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.478094 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.478097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd2bacfe-d11d-431f-bdef-e69bf896271c-tls-certs\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.478331 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.478113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-tmp-dir\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.478331 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.478138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-model-cache\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.478331 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.478172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzpfv\" (UniqueName: \"kubernetes.io/projected/fd2bacfe-d11d-431f-bdef-e69bf896271c-kube-api-access-qzpfv\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.478331 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.478270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-dshm\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.478541 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.478493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-tmp-dir\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.478598 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.478559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-home\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.478638 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.478595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-model-cache\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.480485 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.480467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-dshm\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.481017 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.480995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd2bacfe-d11d-431f-bdef-e69bf896271c-tls-certs\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.486748 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.486726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzpfv\" (UniqueName: \"kubernetes.io/projected/fd2bacfe-d11d-431f-bdef-e69bf896271c-kube-api-access-qzpfv\") pod \"rate-test-model-d90e71-kserve-58d5c99f5-kjh45\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") " pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.561919 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.561892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:35.832157 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.832132    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"]
Sep 11 18:48:35.833967 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:48:35.833940    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd2bacfe_d11d_431f_bdef_e69bf896271c.slice/crio-032b91620cb04de7bfe3eed9822fc43152f8fcd04253a0ada3a6dee80ea0c948 WatchSource:0}: Error finding container 032b91620cb04de7bfe3eed9822fc43152f8fcd04253a0ada3a6dee80ea0c948: Status 404 returned error can't find the container with id 032b91620cb04de7bfe3eed9822fc43152f8fcd04253a0ada3a6dee80ea0c948
Sep 11 18:48:35.938365 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.938332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45" event={"ID":"fd2bacfe-d11d-431f-bdef-e69bf896271c","Type":"ContainerStarted","Data":"c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe"}
Sep 11 18:48:35.938520 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.938373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45" event={"ID":"fd2bacfe-d11d-431f-bdef-e69bf896271c","Type":"ContainerStarted","Data":"032b91620cb04de7bfe3eed9822fc43152f8fcd04253a0ada3a6dee80ea0c948"}
Sep 11 18:48:35.953389 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:35.953339    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45" podStartSLOduration=0.953323722 podStartE2EDuration="953.323722ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:35.95128028 +0000 UTC m=+1980.530544837" watchObservedRunningTime="2026-09-11 18:48:35.953323722 +0000 UTC m=+1980.532588282"
Sep 11 18:48:41.091271 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.091212    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl" podUID="ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" containerName="main" containerID="cri-o://a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83" gracePeriod=45
Sep 11 18:48:41.342368 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.342305    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:48:41.533736 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.533695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tls-certs\") pod \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") "
Sep 11 18:48:41.533923 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.533815    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-model-cache\") pod \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") "
Sep 11 18:48:41.533923 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.533852    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-home\") pod \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") "
Sep 11 18:48:41.533923 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.533888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tmp-dir\") pod \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") "
Sep 11 18:48:41.533923 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.533921    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r55hz\" (UniqueName: \"kubernetes.io/projected/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-kube-api-access-r55hz\") pod \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") "
Sep 11 18:48:41.534164 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.533970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-dshm\") pod \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\" (UID: \"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4\") "
Sep 11 18:48:41.534164 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.534135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-home" (OuterVolumeSpecName: "home") pod "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" (UID: "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:48:41.534259 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.534156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" (UID: "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:48:41.534259 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.534175    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" (UID: "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:48:41.534364 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.534327    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:48:41.534364 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.534347    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:48:41.534437 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.534382    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:48:41.536036 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.536013    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-dshm" (OuterVolumeSpecName: "dshm") pod "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" (UID: "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:48:41.536156 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.536018    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" (UID: "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:48:41.536156 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.536040    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-kube-api-access-r55hz" (OuterVolumeSpecName: "kube-api-access-r55hz") pod "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" (UID: "ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4"). InnerVolumeSpecName "kube-api-access-r55hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:48:41.635916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.635831    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r55hz\" (UniqueName: \"kubernetes.io/projected/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-kube-api-access-r55hz\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:48:41.635916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.635860    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:48:41.635916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.635870    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:48:41.961589 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.961557    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" containerID="a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83" exitCode=0
Sep 11 18:48:41.961745 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.961599    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"
Sep 11 18:48:41.961745 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.961634    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl" event={"ID":"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4","Type":"ContainerDied","Data":"a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83"}
Sep 11 18:48:41.961745 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.961686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl" event={"ID":"ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4","Type":"ContainerDied","Data":"690d1d57293bb64e51d50d43e530e749433a0325920348e49ebc23f83f1a5408"}
Sep 11 18:48:41.961745 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.961705    2578 scope.go:117] "RemoveContainer" containerID="a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83"
Sep 11 18:48:41.970312 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.970295    2578 scope.go:117] "RemoveContainer" containerID="a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83"
Sep 11 18:48:41.970568 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:48:41.970549    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83\": container with ID starting with a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83 not found: ID does not exist" containerID="a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83"
Sep 11 18:48:41.970629 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.970577    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83"} err="failed to get container status \"a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83\": rpc error: code = NotFound desc = could not find container \"a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83\": container with ID starting with a569fe1880bbc836bf4f955fa6729a3478e415633cc2a3e514d3c1d8c91dcc83 not found: ID does not exist"
Sep 11 18:48:41.981779 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.981755    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"]
Sep 11 18:48:41.985208 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:41.985187    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/sub-test-model-d90e71-kserve-bbdd6b95f-pk8jl"]
Sep 11 18:48:43.937328 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:43.937291    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" path="/var/lib/kubelet/pods/ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4/volumes"
Sep 11 18:48:45.562950 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:45.562917    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:45.563307 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:45.562975    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:45.575055 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:45.575030    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:48:45.991922 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:48:45.991838    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:50:05.582246 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:05.582166    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"]
Sep 11 18:50:13.354532 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.354492    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"]
Sep 11 18:50:13.355028 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.354924    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" containerName="main"
Sep 11 18:50:13.355028 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.354939    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" containerName="main"
Sep 11 18:50:13.355028 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.355011    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddbfd5a1-32c3-47ac-9fdf-9fa2998575e4" containerName="main"
Sep 11 18:50:13.358236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.358218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:50:13.365594 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.365569    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-da4d13-dockercfg-xskms\""
Sep 11 18:50:13.365731 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.365599    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-da4d13\""
Sep 11 18:50:13.372337 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.372317    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"]
Sep 11 18:50:13.408040 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.408010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d26fd2-de92-4913-b317-090fdde7e740-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs\" (UID: \"e4d26fd2-de92-4913-b317-090fdde7e740\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:50:13.408173 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.408135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx7zl\" (UniqueName: \"kubernetes.io/projected/e4d26fd2-de92-4913-b317-090fdde7e740-kube-api-access-nx7zl\") pod \"payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs\" (UID: \"e4d26fd2-de92-4913-b317-090fdde7e740\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:50:13.508903 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.508866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx7zl\" (UniqueName: \"kubernetes.io/projected/e4d26fd2-de92-4913-b317-090fdde7e740-kube-api-access-nx7zl\") pod \"payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs\" (UID: \"e4d26fd2-de92-4913-b317-090fdde7e740\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:50:13.509063 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.508936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d26fd2-de92-4913-b317-090fdde7e740-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs\" (UID: \"e4d26fd2-de92-4913-b317-090fdde7e740\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:50:13.509604 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.509586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d26fd2-de92-4913-b317-090fdde7e740-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs\" (UID: \"e4d26fd2-de92-4913-b317-090fdde7e740\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:50:13.518622 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.518598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx7zl\" (UniqueName: \"kubernetes.io/projected/e4d26fd2-de92-4913-b317-090fdde7e740-kube-api-access-nx7zl\") pod \"payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs\" (UID: \"e4d26fd2-de92-4913-b317-090fdde7e740\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:50:13.668558 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.668528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:50:13.945510 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:13.945481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"]
Sep 11 18:50:13.947195 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:50:13.947162    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d26fd2_de92_4913_b317_090fdde7e740.slice/crio-c0061eb4df7251ddb05dbac4debd4cec64d8a373a4b9ed3fae4e0b346453c16c WatchSource:0}: Error finding container c0061eb4df7251ddb05dbac4debd4cec64d8a373a4b9ed3fae4e0b346453c16c: Status 404 returned error can't find the container with id c0061eb4df7251ddb05dbac4debd4cec64d8a373a4b9ed3fae4e0b346453c16c
Sep 11 18:50:14.411696 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:14.411636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs" event={"ID":"e4d26fd2-de92-4913-b317-090fdde7e740","Type":"ContainerStarted","Data":"3f7144be4abb40450d16f3ebb48ff79f79928f7231faa06481d606d970637f87"}
Sep 11 18:50:14.412111 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:14.411811    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:50:14.412111 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:14.411859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs" event={"ID":"e4d26fd2-de92-4913-b317-090fdde7e740","Type":"ContainerStarted","Data":"c0061eb4df7251ddb05dbac4debd4cec64d8a373a4b9ed3fae4e0b346453c16c"}
Sep 11 18:50:14.430247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:14.430200    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs" podStartSLOduration=1.43018571 podStartE2EDuration="1.43018571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:14.429772456 +0000 UTC m=+2079.009037016" watchObservedRunningTime="2026-09-11 18:50:14.43018571 +0000 UTC m=+2079.009450269"
Sep 11 18:50:20.600509 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.600441    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45" podUID="fd2bacfe-d11d-431f-bdef-e69bf896271c" containerName="main" containerID="cri-o://c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe" gracePeriod=45
Sep 11 18:50:20.857811 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.857747    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:50:20.981447 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.981409    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd2bacfe-d11d-431f-bdef-e69bf896271c-tls-certs\") pod \"fd2bacfe-d11d-431f-bdef-e69bf896271c\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") "
Sep 11 18:50:20.981447 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.981448    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzpfv\" (UniqueName: \"kubernetes.io/projected/fd2bacfe-d11d-431f-bdef-e69bf896271c-kube-api-access-qzpfv\") pod \"fd2bacfe-d11d-431f-bdef-e69bf896271c\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") "
Sep 11 18:50:20.981710 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.981547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-home\") pod \"fd2bacfe-d11d-431f-bdef-e69bf896271c\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") "
Sep 11 18:50:20.981710 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.981695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-tmp-dir\") pod \"fd2bacfe-d11d-431f-bdef-e69bf896271c\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") "
Sep 11 18:50:20.981825 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.981736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-dshm\") pod \"fd2bacfe-d11d-431f-bdef-e69bf896271c\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") "
Sep 11 18:50:20.981825 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.981785    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-model-cache\") pod \"fd2bacfe-d11d-431f-bdef-e69bf896271c\" (UID: \"fd2bacfe-d11d-431f-bdef-e69bf896271c\") "
Sep 11 18:50:20.981825 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.981807    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-home" (OuterVolumeSpecName: "home") pod "fd2bacfe-d11d-431f-bdef-e69bf896271c" (UID: "fd2bacfe-d11d-431f-bdef-e69bf896271c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:20.981958 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.981933    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd2bacfe-d11d-431f-bdef-e69bf896271c" (UID: "fd2bacfe-d11d-431f-bdef-e69bf896271c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:20.982065 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.982047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd2bacfe-d11d-431f-bdef-e69bf896271c" (UID: "fd2bacfe-d11d-431f-bdef-e69bf896271c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:20.982178 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.982163    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:50:20.982236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.982197    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:50:20.982236 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.982207    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:50:20.983602 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.983577    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd2bacfe-d11d-431f-bdef-e69bf896271c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd2bacfe-d11d-431f-bdef-e69bf896271c" (UID: "fd2bacfe-d11d-431f-bdef-e69bf896271c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:50:20.983602 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.983588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2bacfe-d11d-431f-bdef-e69bf896271c-kube-api-access-qzpfv" (OuterVolumeSpecName: "kube-api-access-qzpfv") pod "fd2bacfe-d11d-431f-bdef-e69bf896271c" (UID: "fd2bacfe-d11d-431f-bdef-e69bf896271c"). InnerVolumeSpecName "kube-api-access-qzpfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:20.983769 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:20.983719    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-dshm" (OuterVolumeSpecName: "dshm") pod "fd2bacfe-d11d-431f-bdef-e69bf896271c" (UID: "fd2bacfe-d11d-431f-bdef-e69bf896271c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:21.083631 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.083591    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd2bacfe-d11d-431f-bdef-e69bf896271c-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:50:21.083631 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.083620    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd2bacfe-d11d-431f-bdef-e69bf896271c-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:50:21.083631 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.083634    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzpfv\" (UniqueName: \"kubernetes.io/projected/fd2bacfe-d11d-431f-bdef-e69bf896271c-kube-api-access-qzpfv\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:50:21.447796 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.447760    2578 generic.go:356] "Generic (PLEG): container finished" podID="fd2bacfe-d11d-431f-bdef-e69bf896271c" containerID="c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe" exitCode=0
Sep 11 18:50:21.447964 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.447799    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"
Sep 11 18:50:21.447964 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.447837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45" event={"ID":"fd2bacfe-d11d-431f-bdef-e69bf896271c","Type":"ContainerDied","Data":"c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe"}
Sep 11 18:50:21.447964 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.447881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45" event={"ID":"fd2bacfe-d11d-431f-bdef-e69bf896271c","Type":"ContainerDied","Data":"032b91620cb04de7bfe3eed9822fc43152f8fcd04253a0ada3a6dee80ea0c948"}
Sep 11 18:50:21.447964 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.447902    2578 scope.go:117] "RemoveContainer" containerID="c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe"
Sep 11 18:50:21.456803 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.456785    2578 scope.go:117] "RemoveContainer" containerID="c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe"
Sep 11 18:50:21.457057 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:50:21.457040    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe\": container with ID starting with c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe not found: ID does not exist" containerID="c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe"
Sep 11 18:50:21.457105 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.457064    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe"} err="failed to get container status \"c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe\": rpc error: code = NotFound desc = could not find container \"c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe\": container with ID starting with c42dfe71d731227c0b13a4b01dcef7cff30adaa303bf93113c78ee678c119ebe not found: ID does not exist"
Sep 11 18:50:21.469363 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.469341    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"]
Sep 11 18:50:21.473390 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.473371    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d90e71/rate-test-model-d90e71-kserve-58d5c99f5-kjh45"]
Sep 11 18:50:21.937492 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:21.937456    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd2bacfe-d11d-431f-bdef-e69bf896271c" path="/var/lib/kubelet/pods/fd2bacfe-d11d-431f-bdef-e69bf896271c/volumes"
Sep 11 18:50:25.419642 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:25.419613    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:50:43.166910 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.166878    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"]
Sep 11 18:50:43.167340 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.167268    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2bacfe-d11d-431f-bdef-e69bf896271c" containerName="main"
Sep 11 18:50:43.167340 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.167278    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2bacfe-d11d-431f-bdef-e69bf896271c" containerName="main"
Sep 11 18:50:43.167425 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.167354    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2bacfe-d11d-431f-bdef-e69bf896271c" containerName="main"
Sep 11 18:50:43.170280 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.170260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:50:43.173339 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.173318    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0ed2c8\""
Sep 11 18:50:43.173465 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.173325    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0ed2c8-dockercfg-tp6cs\""
Sep 11 18:50:43.176844 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.176822    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"]
Sep 11 18:50:43.180418 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.180401    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:50:43.184567 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.184548    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"]
Sep 11 18:50:43.194842 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.194823    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"]
Sep 11 18:50:43.196596 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.196571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkdlm\" (UniqueName: \"kubernetes.io/projected/4fb14558-59a5-4474-8f52-7781ca63c8f0-kube-api-access-fkdlm\") pod \"payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs\" (UID: \"4fb14558-59a5-4474-8f52-7781ca63c8f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:50:43.196709 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.196641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb14558-59a5-4474-8f52-7781ca63c8f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs\" (UID: \"4fb14558-59a5-4474-8f52-7781ca63c8f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:50:43.196767 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.196748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgn8f\" (UniqueName: \"kubernetes.io/projected/4b2f6c27-b605-4e7f-9976-f9d50786a112-kube-api-access-dgn8f\") pod \"payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9\" (UID: \"4b2f6c27-b605-4e7f-9976-f9d50786a112\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:50:43.196809 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.196783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2f6c27-b605-4e7f-9976-f9d50786a112-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9\" (UID: \"4b2f6c27-b605-4e7f-9976-f9d50786a112\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:50:43.297701 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.297631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2f6c27-b605-4e7f-9976-f9d50786a112-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9\" (UID: \"4b2f6c27-b605-4e7f-9976-f9d50786a112\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:50:43.297891 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.297859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkdlm\" (UniqueName: \"kubernetes.io/projected/4fb14558-59a5-4474-8f52-7781ca63c8f0-kube-api-access-fkdlm\") pod \"payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs\" (UID: \"4fb14558-59a5-4474-8f52-7781ca63c8f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:50:43.297964 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.297945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb14558-59a5-4474-8f52-7781ca63c8f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs\" (UID: \"4fb14558-59a5-4474-8f52-7781ca63c8f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:50:43.298055 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.298039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgn8f\" (UniqueName: \"kubernetes.io/projected/4b2f6c27-b605-4e7f-9976-f9d50786a112-kube-api-access-dgn8f\") pod \"payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9\" (UID: \"4b2f6c27-b605-4e7f-9976-f9d50786a112\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:50:43.298793 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.298767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2f6c27-b605-4e7f-9976-f9d50786a112-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9\" (UID: \"4b2f6c27-b605-4e7f-9976-f9d50786a112\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:50:43.299023 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.298997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb14558-59a5-4474-8f52-7781ca63c8f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs\" (UID: \"4fb14558-59a5-4474-8f52-7781ca63c8f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:50:43.306636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.306611    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgn8f\" (UniqueName: \"kubernetes.io/projected/4b2f6c27-b605-4e7f-9976-f9d50786a112-kube-api-access-dgn8f\") pod \"payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9\" (UID: \"4b2f6c27-b605-4e7f-9976-f9d50786a112\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:50:43.306750 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.306637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkdlm\" (UniqueName: \"kubernetes.io/projected/4fb14558-59a5-4474-8f52-7781ca63c8f0-kube-api-access-fkdlm\") pod \"payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs\" (UID: \"4fb14558-59a5-4474-8f52-7781ca63c8f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:50:43.480685 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.480561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:50:43.491821 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.491793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:50:43.762953 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.762918    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"]
Sep 11 18:50:43.764695 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:50:43.764666    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb14558_59a5_4474_8f52_7781ca63c8f0.slice/crio-7b8b8bd0aa6a50b04606d03e31770fff5040bb91915a0764b31c70006c81246d WatchSource:0}: Error finding container 7b8b8bd0aa6a50b04606d03e31770fff5040bb91915a0764b31c70006c81246d: Status 404 returned error can't find the container with id 7b8b8bd0aa6a50b04606d03e31770fff5040bb91915a0764b31c70006c81246d
Sep 11 18:50:43.769691 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:43.769429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"]
Sep 11 18:50:43.771687 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:50:43.770966    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2f6c27_b605_4e7f_9976_f9d50786a112.slice/crio-a9afb8734446a369c71d59745af4c800e17038b0bbc1675aba8c4f05451b6fea WatchSource:0}: Error finding container a9afb8734446a369c71d59745af4c800e17038b0bbc1675aba8c4f05451b6fea: Status 404 returned error can't find the container with id a9afb8734446a369c71d59745af4c800e17038b0bbc1675aba8c4f05451b6fea
Sep 11 18:50:44.575336 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:44.575303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9" event={"ID":"4b2f6c27-b605-4e7f-9976-f9d50786a112","Type":"ContainerStarted","Data":"f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad"}
Sep 11 18:50:44.575838 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:44.575354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9" event={"ID":"4b2f6c27-b605-4e7f-9976-f9d50786a112","Type":"ContainerStarted","Data":"a9afb8734446a369c71d59745af4c800e17038b0bbc1675aba8c4f05451b6fea"}
Sep 11 18:50:44.575838 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:44.575501    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:50:44.577273 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:44.577248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs" event={"ID":"4fb14558-59a5-4474-8f52-7781ca63c8f0","Type":"ContainerStarted","Data":"637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f"}
Sep 11 18:50:44.577405 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:44.577294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs" event={"ID":"4fb14558-59a5-4474-8f52-7781ca63c8f0","Type":"ContainerStarted","Data":"7b8b8bd0aa6a50b04606d03e31770fff5040bb91915a0764b31c70006c81246d"}
Sep 11 18:50:44.577405 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:44.577324    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:50:44.596685 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:44.595238    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9" podStartSLOduration=1.595219246 podStartE2EDuration="1.595219246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:44.591817197 +0000 UTC m=+2109.171081757" watchObservedRunningTime="2026-09-11 18:50:44.595219246 +0000 UTC m=+2109.174483804"
Sep 11 18:50:44.609845 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:44.609795    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs" podStartSLOduration=1.6097791350000001 podStartE2EDuration="1.609779135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:44.609675643 +0000 UTC m=+2109.188940200" watchObservedRunningTime="2026-09-11 18:50:44.609779135 +0000 UTC m=+2109.189043698"
Sep 11 18:50:55.584921 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:55.584884    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:50:55.585333 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:50:55.584999    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:51:09.610001 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:09.609970    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9_4b2f6c27-b605-4e7f-9976-f9d50786a112/payload-processing/0.log"
Sep 11 18:51:25.651777 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.651737    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"]
Sep 11 18:51:25.655389 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.655372    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.658486 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.658464    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-da4d13\"/\"openshift-service-ca.crt\""
Sep 11 18:51:25.659906 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.659889    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-da4d13\"/\"default-dockercfg-7fh9f\""
Sep 11 18:51:25.660019 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.659932    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-da4d13\"/\"kube-root-ca.crt\""
Sep 11 18:51:25.660019 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.659958    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-da4d13\"/\"ipp-route-da4d13-kserve-self-signed-certs\""
Sep 11 18:51:25.664809 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.664789    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"]
Sep 11 18:51:25.798010 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.797976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-dshm\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.798188 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.798037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-model-cache\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.798188 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.798108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-home\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.798188 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.798125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tmp-dir\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.798188 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.798141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tls-certs\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.798333 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.798202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltxxd\" (UniqueName: \"kubernetes.io/projected/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-kube-api-access-ltxxd\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.899752 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.899700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-model-cache\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.899914 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.899784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-home\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.899914 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.899805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tmp-dir\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.899914 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.899821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tls-certs\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.899914 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.899866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltxxd\" (UniqueName: \"kubernetes.io/projected/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-kube-api-access-ltxxd\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.899914 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.899888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-dshm\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.900208 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.900183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-model-cache\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.900286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.900206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-home\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.900286 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.900268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tmp-dir\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.902161 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.902109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-dshm\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.902300 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.902281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tls-certs\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.908044 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.908021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltxxd\" (UniqueName: \"kubernetes.io/projected/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-kube-api-access-ltxxd\") pod \"ipp-route-da4d13-kserve-66cbf794c9-zph8t\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") " pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:25.965924 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:25.965884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:26.243303 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:26.243276    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"]
Sep 11 18:51:26.245523 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:51:26.245488    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1db525e_20cb_438f_87d0_7d8b8b3cd6ea.slice/crio-708c249c3fde17e41dd5eba58ec4debeb7fe4c19465f3605135eac322f0a6ffc WatchSource:0}: Error finding container 708c249c3fde17e41dd5eba58ec4debeb7fe4c19465f3605135eac322f0a6ffc: Status 404 returned error can't find the container with id 708c249c3fde17e41dd5eba58ec4debeb7fe4c19465f3605135eac322f0a6ffc
Sep 11 18:51:26.780124 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:26.780081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t" event={"ID":"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea","Type":"ContainerStarted","Data":"272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142"}
Sep 11 18:51:26.780498 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:26.780131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t" event={"ID":"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea","Type":"ContainerStarted","Data":"708c249c3fde17e41dd5eba58ec4debeb7fe4c19465f3605135eac322f0a6ffc"}
Sep 11 18:51:26.800021 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:26.799962    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t" podStartSLOduration=1.7999457429999999 podStartE2EDuration="1.799945743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:51:26.797997057 +0000 UTC m=+2151.377261618" watchObservedRunningTime="2026-09-11 18:51:26.799945743 +0000 UTC m=+2151.379210302"
Sep 11 18:51:35.966680 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:35.966568    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:35.966680 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:35.966635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:35.979239 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:35.979206    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:36.217964 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:36.217876    2578 scope.go:117] "RemoveContainer" containerID="fe91dab2f5e9a819b76b32f1d826196b3ad3bb3fde28668d81b618332974ca6d"
Sep 11 18:51:36.226808 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:36.226787    2578 scope.go:117] "RemoveContainer" containerID="4766d1e79b1d1c6eb7482eade56204d39200d29db3b34924ca07b3a8b82d15f5"
Sep 11 18:51:36.234724 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:36.234700    2578 scope.go:117] "RemoveContainer" containerID="bac98ce7259dbf87465b818b1fe6d9bd46f40d636f2b9a66d117dfca94044260"
Sep 11 18:51:36.840905 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:36.840875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:51:50.750448 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:51:50.750421    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9_4b2f6c27-b605-4e7f-9976-f9d50786a112/payload-processing/0.log"
Sep 11 18:53:04.006709 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.006625    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"]
Sep 11 18:53:04.007244 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.006849    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs" podUID="e4d26fd2-de92-4913-b317-090fdde7e740" containerName="payload-pre-processing" containerID="cri-o://3f7144be4abb40450d16f3ebb48ff79f79928f7231faa06481d606d970637f87" gracePeriod=30
Sep 11 18:53:04.243295 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.243259    2578 generic.go:356] "Generic (PLEG): container finished" podID="e4d26fd2-de92-4913-b317-090fdde7e740" containerID="3f7144be4abb40450d16f3ebb48ff79f79928f7231faa06481d606d970637f87" exitCode=0
Sep 11 18:53:04.243424 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.243332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs" event={"ID":"e4d26fd2-de92-4913-b317-090fdde7e740","Type":"ContainerDied","Data":"3f7144be4abb40450d16f3ebb48ff79f79928f7231faa06481d606d970637f87"}
Sep 11 18:53:04.243424 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.243405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs" event={"ID":"e4d26fd2-de92-4913-b317-090fdde7e740","Type":"ContainerDied","Data":"c0061eb4df7251ddb05dbac4debd4cec64d8a373a4b9ed3fae4e0b346453c16c"}
Sep 11 18:53:04.243512 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.243428    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0061eb4df7251ddb05dbac4debd4cec64d8a373a4b9ed3fae4e0b346453c16c"
Sep 11 18:53:04.254355 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.254333    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:53:04.340449 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.340413    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d26fd2-de92-4913-b317-090fdde7e740-plugins-config-volume\") pod \"e4d26fd2-de92-4913-b317-090fdde7e740\" (UID: \"e4d26fd2-de92-4913-b317-090fdde7e740\") "
Sep 11 18:53:04.340620 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.340486    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx7zl\" (UniqueName: \"kubernetes.io/projected/e4d26fd2-de92-4913-b317-090fdde7e740-kube-api-access-nx7zl\") pod \"e4d26fd2-de92-4913-b317-090fdde7e740\" (UID: \"e4d26fd2-de92-4913-b317-090fdde7e740\") "
Sep 11 18:53:04.340833 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.340809    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d26fd2-de92-4913-b317-090fdde7e740-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4d26fd2-de92-4913-b317-090fdde7e740" (UID: "e4d26fd2-de92-4913-b317-090fdde7e740"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:53:04.342555 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.342536    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d26fd2-de92-4913-b317-090fdde7e740-kube-api-access-nx7zl" (OuterVolumeSpecName: "kube-api-access-nx7zl") pod "e4d26fd2-de92-4913-b317-090fdde7e740" (UID: "e4d26fd2-de92-4913-b317-090fdde7e740"). InnerVolumeSpecName "kube-api-access-nx7zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:53:04.441885 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.441850    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx7zl\" (UniqueName: \"kubernetes.io/projected/e4d26fd2-de92-4913-b317-090fdde7e740-kube-api-access-nx7zl\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:53:04.441885 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:04.441878    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d26fd2-de92-4913-b317-090fdde7e740-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:53:05.248537 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:05.248505    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"
Sep 11 18:53:05.269990 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:05.269959    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"]
Sep 11 18:53:05.274513 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:05.274482    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da4d13-558b545d98-hkvjs"]
Sep 11 18:53:05.937487 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:05.937458    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4d26fd2-de92-4913-b317-090fdde7e740" path="/var/lib/kubelet/pods/e4d26fd2-de92-4913-b317-090fdde7e740/volumes"
Sep 11 18:53:09.563512 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:09.563480    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"]
Sep 11 18:53:24.582389 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.582341    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t" podUID="c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" containerName="main" containerID="cri-o://272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142" gracePeriod=45
Sep 11 18:53:24.823031 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.823006    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:53:24.928050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.927958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-home\") pod \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") "
Sep 11 18:53:24.928050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.928002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tmp-dir\") pod \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") "
Sep 11 18:53:24.928050 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.928050    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltxxd\" (UniqueName: \"kubernetes.io/projected/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-kube-api-access-ltxxd\") pod \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") "
Sep 11 18:53:24.928333 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.928084    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-dshm\") pod \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") "
Sep 11 18:53:24.928333 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.928122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tls-certs\") pod \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") "
Sep 11 18:53:24.928333 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.928247    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-model-cache\") pod \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\" (UID: \"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea\") "
Sep 11 18:53:24.928333 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.928267    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-home" (OuterVolumeSpecName: "home") pod "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" (UID: "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:53:24.928511 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.928334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" (UID: "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:53:24.928990 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.928946    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" (UID: "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:53:24.930695 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.930344    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:53:24.930695 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.930388    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:53:24.930695 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.930494    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:53:24.931474 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.931443    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-kube-api-access-ltxxd" (OuterVolumeSpecName: "kube-api-access-ltxxd") pod "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" (UID: "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea"). InnerVolumeSpecName "kube-api-access-ltxxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:53:24.936767 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.931980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-dshm" (OuterVolumeSpecName: "dshm") pod "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" (UID: "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:53:24.937903 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:24.937871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" (UID: "c1db525e-20cb-438f-87d0-7d8b8b3cd6ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:53:25.031274 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.031244    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltxxd\" (UniqueName: \"kubernetes.io/projected/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-kube-api-access-ltxxd\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:53:25.031274 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.031271    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:53:25.031274 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.031281    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:53:25.345359 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.345322    2578 generic.go:356] "Generic (PLEG): container finished" podID="c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" containerID="272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142" exitCode=0
Sep 11 18:53:25.345534 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.345379    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"
Sep 11 18:53:25.345534 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.345377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t" event={"ID":"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea","Type":"ContainerDied","Data":"272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142"}
Sep 11 18:53:25.345534 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.345484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t" event={"ID":"c1db525e-20cb-438f-87d0-7d8b8b3cd6ea","Type":"ContainerDied","Data":"708c249c3fde17e41dd5eba58ec4debeb7fe4c19465f3605135eac322f0a6ffc"}
Sep 11 18:53:25.345534 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.345501    2578 scope.go:117] "RemoveContainer" containerID="272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142"
Sep 11 18:53:25.354401 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.354382    2578 scope.go:117] "RemoveContainer" containerID="272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142"
Sep 11 18:53:25.354681 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:53:25.354644    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142\": container with ID starting with 272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142 not found: ID does not exist" containerID="272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142"
Sep 11 18:53:25.354748 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.354689    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142"} err="failed to get container status \"272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142\": rpc error: code = NotFound desc = could not find container \"272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142\": container with ID starting with 272f7ea47cdc3c764ad1d0d511c6cee6a884a644720864e0d38f74de6fb5a142 not found: ID does not exist"
Sep 11 18:53:25.383282 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.383253    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"]
Sep 11 18:53:25.393409 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.393389    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-da4d13/ipp-route-da4d13-kserve-66cbf794c9-zph8t"]
Sep 11 18:53:25.937824 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:53:25.937791    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" path="/var/lib/kubelet/pods/c1db525e-20cb-438f-87d0-7d8b8b3cd6ea/volumes"
Sep 11 18:54:09.877410 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:09.877375    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"]
Sep 11 18:54:09.878012 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:09.877599    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9" podUID="4b2f6c27-b605-4e7f-9976-f9d50786a112" containerName="payload-processing" containerID="cri-o://f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad" gracePeriod=30
Sep 11 18:54:09.887056 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:09.887001    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"]
Sep 11 18:54:09.887434 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:09.887384    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs" podUID="4fb14558-59a5-4474-8f52-7781ca63c8f0" containerName="payload-pre-processing" containerID="cri-o://637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f" gracePeriod=30
Sep 11 18:54:10.163455 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.163425    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:54:10.166679 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.166635    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:54:10.247989 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.247958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2f6c27-b605-4e7f-9976-f9d50786a112-plugins-config-volume\") pod \"4b2f6c27-b605-4e7f-9976-f9d50786a112\" (UID: \"4b2f6c27-b605-4e7f-9976-f9d50786a112\") "
Sep 11 18:54:10.248170 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.248001    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb14558-59a5-4474-8f52-7781ca63c8f0-plugins-config-volume\") pod \"4fb14558-59a5-4474-8f52-7781ca63c8f0\" (UID: \"4fb14558-59a5-4474-8f52-7781ca63c8f0\") "
Sep 11 18:54:10.248170 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.248039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkdlm\" (UniqueName: \"kubernetes.io/projected/4fb14558-59a5-4474-8f52-7781ca63c8f0-kube-api-access-fkdlm\") pod \"4fb14558-59a5-4474-8f52-7781ca63c8f0\" (UID: \"4fb14558-59a5-4474-8f52-7781ca63c8f0\") "
Sep 11 18:54:10.248295 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.248173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgn8f\" (UniqueName: \"kubernetes.io/projected/4b2f6c27-b605-4e7f-9976-f9d50786a112-kube-api-access-dgn8f\") pod \"4b2f6c27-b605-4e7f-9976-f9d50786a112\" (UID: \"4b2f6c27-b605-4e7f-9976-f9d50786a112\") "
Sep 11 18:54:10.248438 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.248410    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb14558-59a5-4474-8f52-7781ca63c8f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fb14558-59a5-4474-8f52-7781ca63c8f0" (UID: "4fb14558-59a5-4474-8f52-7781ca63c8f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:54:10.248438 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.248414    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2f6c27-b605-4e7f-9976-f9d50786a112-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b2f6c27-b605-4e7f-9976-f9d50786a112" (UID: "4b2f6c27-b605-4e7f-9976-f9d50786a112"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:54:10.248624 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.248508    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2f6c27-b605-4e7f-9976-f9d50786a112-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:10.248624 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.248529    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb14558-59a5-4474-8f52-7781ca63c8f0-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:10.250226 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.250204    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb14558-59a5-4474-8f52-7781ca63c8f0-kube-api-access-fkdlm" (OuterVolumeSpecName: "kube-api-access-fkdlm") pod "4fb14558-59a5-4474-8f52-7781ca63c8f0" (UID: "4fb14558-59a5-4474-8f52-7781ca63c8f0"). InnerVolumeSpecName "kube-api-access-fkdlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:54:10.250316 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.250271    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2f6c27-b605-4e7f-9976-f9d50786a112-kube-api-access-dgn8f" (OuterVolumeSpecName: "kube-api-access-dgn8f") pod "4b2f6c27-b605-4e7f-9976-f9d50786a112" (UID: "4b2f6c27-b605-4e7f-9976-f9d50786a112"). InnerVolumeSpecName "kube-api-access-dgn8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:54:10.349809 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.349771    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgn8f\" (UniqueName: \"kubernetes.io/projected/4b2f6c27-b605-4e7f-9976-f9d50786a112-kube-api-access-dgn8f\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:10.349809 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.349801    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkdlm\" (UniqueName: \"kubernetes.io/projected/4fb14558-59a5-4474-8f52-7781ca63c8f0-kube-api-access-fkdlm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:10.560266 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.560231    2578 generic.go:356] "Generic (PLEG): container finished" podID="4b2f6c27-b605-4e7f-9976-f9d50786a112" containerID="f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad" exitCode=0
Sep 11 18:54:10.560450 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.560272    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"
Sep 11 18:54:10.560450 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.560327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9" event={"ID":"4b2f6c27-b605-4e7f-9976-f9d50786a112","Type":"ContainerDied","Data":"f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad"}
Sep 11 18:54:10.560450 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.560379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9" event={"ID":"4b2f6c27-b605-4e7f-9976-f9d50786a112","Type":"ContainerDied","Data":"a9afb8734446a369c71d59745af4c800e17038b0bbc1675aba8c4f05451b6fea"}
Sep 11 18:54:10.560450 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.560401    2578 scope.go:117] "RemoveContainer" containerID="f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad"
Sep 11 18:54:10.561728 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.561708    2578 generic.go:356] "Generic (PLEG): container finished" podID="4fb14558-59a5-4474-8f52-7781ca63c8f0" containerID="637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f" exitCode=0
Sep 11 18:54:10.561838 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.561751    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"
Sep 11 18:54:10.561838 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.561784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs" event={"ID":"4fb14558-59a5-4474-8f52-7781ca63c8f0","Type":"ContainerDied","Data":"637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f"}
Sep 11 18:54:10.561838 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.561831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs" event={"ID":"4fb14558-59a5-4474-8f52-7781ca63c8f0","Type":"ContainerDied","Data":"7b8b8bd0aa6a50b04606d03e31770fff5040bb91915a0764b31c70006c81246d"}
Sep 11 18:54:10.568994 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.568975    2578 scope.go:117] "RemoveContainer" containerID="f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad"
Sep 11 18:54:10.569235 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:54:10.569218    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad\": container with ID starting with f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad not found: ID does not exist" containerID="f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad"
Sep 11 18:54:10.569293 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.569241    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad"} err="failed to get container status \"f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad\": rpc error: code = NotFound desc = could not find container \"f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad\": container with ID starting with f1b486e6f5bf406b147de2491f2da7eea7a988c4776401e3a4f3d9d935c9c3ad not found: ID does not exist"
Sep 11 18:54:10.569293 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.569257    2578 scope.go:117] "RemoveContainer" containerID="637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f"
Sep 11 18:54:10.578279 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.578265    2578 scope.go:117] "RemoveContainer" containerID="637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f"
Sep 11 18:54:10.578534 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:54:10.578515    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f\": container with ID starting with 637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f not found: ID does not exist" containerID="637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f"
Sep 11 18:54:10.578572 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.578542    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f"} err="failed to get container status \"637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f\": rpc error: code = NotFound desc = could not find container \"637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f\": container with ID starting with 637bbca922caf213f48ba0fe814edfd61e6855ce2a1a464cc0b77c8d7d70c66f not found: ID does not exist"
Sep 11 18:54:10.588190 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.588159    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"]
Sep 11 18:54:10.590210 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.590188    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ed2c8-6785c84846-jd7hs"]
Sep 11 18:54:10.601131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.601108    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"]
Sep 11 18:54:10.604047 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:10.604025    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ed2c8-55485dc4f5-mr7v9"]
Sep 11 18:54:11.937822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:11.937789    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2f6c27-b605-4e7f-9976-f9d50786a112" path="/var/lib/kubelet/pods/4b2f6c27-b605-4e7f-9976-f9d50786a112/volumes"
Sep 11 18:54:11.938195 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:11.938172    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fb14558-59a5-4474-8f52-7781ca63c8f0" path="/var/lib/kubelet/pods/4fb14558-59a5-4474-8f52-7781ca63c8f0/volumes"
Sep 11 18:54:27.778692 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:27.778644    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"]
Sep 11 18:54:27.779095 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:27.778879    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6" podUID="636e0fa1-283e-4e81-96b8-070852063a4b" containerName="payload-processing" containerID="cri-o://c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d" gracePeriod=30
Sep 11 18:54:27.787478 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:27.787449    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"]
Sep 11 18:54:27.787747 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:27.787700    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4" podUID="a544c074-ed04-40a5-b41b-93f8cfbbfb66" containerName="payload-pre-processing" containerID="cri-o://7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974" gracePeriod=30
Sep 11 18:54:28.083946 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.083914    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:54:28.087381 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.087358    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:54:28.223543 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.223513    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsqn8\" (UniqueName: \"kubernetes.io/projected/a544c074-ed04-40a5-b41b-93f8cfbbfb66-kube-api-access-hsqn8\") pod \"a544c074-ed04-40a5-b41b-93f8cfbbfb66\" (UID: \"a544c074-ed04-40a5-b41b-93f8cfbbfb66\") "
Sep 11 18:54:28.223543 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.223545    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qr5h\" (UniqueName: \"kubernetes.io/projected/636e0fa1-283e-4e81-96b8-070852063a4b-kube-api-access-5qr5h\") pod \"636e0fa1-283e-4e81-96b8-070852063a4b\" (UID: \"636e0fa1-283e-4e81-96b8-070852063a4b\") "
Sep 11 18:54:28.223765 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.223603    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636e0fa1-283e-4e81-96b8-070852063a4b-plugins-config-volume\") pod \"636e0fa1-283e-4e81-96b8-070852063a4b\" (UID: \"636e0fa1-283e-4e81-96b8-070852063a4b\") "
Sep 11 18:54:28.223765 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.223631    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a544c074-ed04-40a5-b41b-93f8cfbbfb66-plugins-config-volume\") pod \"a544c074-ed04-40a5-b41b-93f8cfbbfb66\" (UID: \"a544c074-ed04-40a5-b41b-93f8cfbbfb66\") "
Sep 11 18:54:28.224056 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.224032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636e0fa1-283e-4e81-96b8-070852063a4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "636e0fa1-283e-4e81-96b8-070852063a4b" (UID: "636e0fa1-283e-4e81-96b8-070852063a4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:54:28.224155 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.224061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a544c074-ed04-40a5-b41b-93f8cfbbfb66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a544c074-ed04-40a5-b41b-93f8cfbbfb66" (UID: "a544c074-ed04-40a5-b41b-93f8cfbbfb66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:54:28.225626 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.225600    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a544c074-ed04-40a5-b41b-93f8cfbbfb66-kube-api-access-hsqn8" (OuterVolumeSpecName: "kube-api-access-hsqn8") pod "a544c074-ed04-40a5-b41b-93f8cfbbfb66" (UID: "a544c074-ed04-40a5-b41b-93f8cfbbfb66"). InnerVolumeSpecName "kube-api-access-hsqn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:54:28.225626 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.225622    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636e0fa1-283e-4e81-96b8-070852063a4b-kube-api-access-5qr5h" (OuterVolumeSpecName: "kube-api-access-5qr5h") pod "636e0fa1-283e-4e81-96b8-070852063a4b" (UID: "636e0fa1-283e-4e81-96b8-070852063a4b"). InnerVolumeSpecName "kube-api-access-5qr5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:54:28.325131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.325039    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsqn8\" (UniqueName: \"kubernetes.io/projected/a544c074-ed04-40a5-b41b-93f8cfbbfb66-kube-api-access-hsqn8\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:28.325131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.325074    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qr5h\" (UniqueName: \"kubernetes.io/projected/636e0fa1-283e-4e81-96b8-070852063a4b-kube-api-access-5qr5h\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:28.325131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.325084    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636e0fa1-283e-4e81-96b8-070852063a4b-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:28.325131 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.325092    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a544c074-ed04-40a5-b41b-93f8cfbbfb66-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:28.653965 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.653877    2578 generic.go:356] "Generic (PLEG): container finished" podID="636e0fa1-283e-4e81-96b8-070852063a4b" containerID="c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d" exitCode=0
Sep 11 18:54:28.653965 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.653923    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"
Sep 11 18:54:28.654255 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.653966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6" event={"ID":"636e0fa1-283e-4e81-96b8-070852063a4b","Type":"ContainerDied","Data":"c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d"}
Sep 11 18:54:28.654255 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.654015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6" event={"ID":"636e0fa1-283e-4e81-96b8-070852063a4b","Type":"ContainerDied","Data":"ac7bf1eac0624ca52e223bd9050b2bd27edb94c0dc2675812297183031a96b59"}
Sep 11 18:54:28.654255 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.654034    2578 scope.go:117] "RemoveContainer" containerID="c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d"
Sep 11 18:54:28.655390 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.655353    2578 generic.go:356] "Generic (PLEG): container finished" podID="a544c074-ed04-40a5-b41b-93f8cfbbfb66" containerID="7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974" exitCode=0
Sep 11 18:54:28.655390 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.655382    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"
Sep 11 18:54:28.655531 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.655412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4" event={"ID":"a544c074-ed04-40a5-b41b-93f8cfbbfb66","Type":"ContainerDied","Data":"7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974"}
Sep 11 18:54:28.655531 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.655455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4" event={"ID":"a544c074-ed04-40a5-b41b-93f8cfbbfb66","Type":"ContainerDied","Data":"c3c60514a3cb9ad175c72d7bd5a0b80409350632d9004d224ffb80e7da1e01b6"}
Sep 11 18:54:28.663546 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.663528    2578 scope.go:117] "RemoveContainer" containerID="c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d"
Sep 11 18:54:28.663884 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:54:28.663852    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d\": container with ID starting with c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d not found: ID does not exist" containerID="c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d"
Sep 11 18:54:28.663992 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.663893    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d"} err="failed to get container status \"c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d\": rpc error: code = NotFound desc = could not find container \"c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d\": container with ID starting with c2cffbcafe08c89c3c2b049f7d81cf219785887dd0a49b418e677fb3b299686d not found: ID does not exist"
Sep 11 18:54:28.663992 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.663915    2578 scope.go:117] "RemoveContainer" containerID="7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974"
Sep 11 18:54:28.671979 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.671965    2578 scope.go:117] "RemoveContainer" containerID="7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974"
Sep 11 18:54:28.672227 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:54:28.672206    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974\": container with ID starting with 7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974 not found: ID does not exist" containerID="7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974"
Sep 11 18:54:28.672308 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.672231    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974"} err="failed to get container status \"7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974\": rpc error: code = NotFound desc = could not find container \"7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974\": container with ID starting with 7e1b04b91ed72efa8073d5de80b14aa8f8d71500b1099c0880d39bfeefa1d974 not found: ID does not exist"
Sep 11 18:54:28.680895 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.680872    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"]
Sep 11 18:54:28.683367 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.683347    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d90e71-75cf48b6b6-s4nj4"]
Sep 11 18:54:28.696667 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.696615    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"]
Sep 11 18:54:28.700882 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:28.700858    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d90e71-64c7958767-p9dg6"]
Sep 11 18:54:29.938319 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:29.938282    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="636e0fa1-283e-4e81-96b8-070852063a4b" path="/var/lib/kubelet/pods/636e0fa1-283e-4e81-96b8-070852063a4b/volumes"
Sep 11 18:54:29.938742 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:29.938726    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a544c074-ed04-40a5-b41b-93f8cfbbfb66" path="/var/lib/kubelet/pods/a544c074-ed04-40a5-b41b-93f8cfbbfb66/volumes"
Sep 11 18:54:45.556940 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.556901    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"]
Sep 11 18:54:45.557524 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.557125    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp" podUID="5ea330cc-51eb-4a91-b987-1c70adf4e896" containerName="maas-api" containerID="cri-o://4debfd7714eee2b07beb90681112db67cdaf6b7eced2290ee1d66a76a54f3537" gracePeriod=30
Sep 11 18:54:45.624387 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.624352    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"]
Sep 11 18:54:45.624736 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.624619    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d" podUID="bbcd7c45-75d7-43eb-a887-4eace477ec09" containerName="payload-pre-processing" containerID="cri-o://f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8" gracePeriod=30
Sep 11 18:54:45.740394 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.740362    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ea330cc-51eb-4a91-b987-1c70adf4e896" containerID="4debfd7714eee2b07beb90681112db67cdaf6b7eced2290ee1d66a76a54f3537" exitCode=0
Sep 11 18:54:45.740558 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.740440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp" event={"ID":"5ea330cc-51eb-4a91-b987-1c70adf4e896","Type":"ContainerDied","Data":"4debfd7714eee2b07beb90681112db67cdaf6b7eced2290ee1d66a76a54f3537"}
Sep 11 18:54:45.780869 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.780821    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d" podUID="bbcd7c45-75d7-43eb-a887-4eace477ec09" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.62:9004: connect: connection refused"
Sep 11 18:54:45.780869 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.780843    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp" podUID="5ea330cc-51eb-4a91-b987-1c70adf4e896" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": dial tcp 10.133.0.61:8443: connect: connection refused"
Sep 11 18:54:45.862575 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.862548    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:54:45.896381 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.896359    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:54:45.994266 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.994230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-metrics-certs\") pod \"5ea330cc-51eb-4a91-b987-1c70adf4e896\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") "
Sep 11 18:54:45.994266 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.994267    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-maas-api-tls\") pod \"5ea330cc-51eb-4a91-b987-1c70adf4e896\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") "
Sep 11 18:54:45.994521 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.994295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea330cc-51eb-4a91-b987-1c70adf4e896-openshift-service-ca\") pod \"5ea330cc-51eb-4a91-b987-1c70adf4e896\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") "
Sep 11 18:54:45.994521 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.994322    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vz47\" (UniqueName: \"kubernetes.io/projected/5ea330cc-51eb-4a91-b987-1c70adf4e896-kube-api-access-6vz47\") pod \"5ea330cc-51eb-4a91-b987-1c70adf4e896\" (UID: \"5ea330cc-51eb-4a91-b987-1c70adf4e896\") "
Sep 11 18:54:45.994730 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.994705    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea330cc-51eb-4a91-b987-1c70adf4e896-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ea330cc-51eb-4a91-b987-1c70adf4e896" (UID: "5ea330cc-51eb-4a91-b987-1c70adf4e896"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:54:45.996435 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.996401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ea330cc-51eb-4a91-b987-1c70adf4e896" (UID: "5ea330cc-51eb-4a91-b987-1c70adf4e896"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:54:45.996548 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.996441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5ea330cc-51eb-4a91-b987-1c70adf4e896" (UID: "5ea330cc-51eb-4a91-b987-1c70adf4e896"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:54:45.996548 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:45.996440    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea330cc-51eb-4a91-b987-1c70adf4e896-kube-api-access-6vz47" (OuterVolumeSpecName: "kube-api-access-6vz47") pod "5ea330cc-51eb-4a91-b987-1c70adf4e896" (UID: "5ea330cc-51eb-4a91-b987-1c70adf4e896"). InnerVolumeSpecName "kube-api-access-6vz47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:54:46.095848 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.095749    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbcd7c45-75d7-43eb-a887-4eace477ec09-plugins-config-volume\") pod \"bbcd7c45-75d7-43eb-a887-4eace477ec09\" (UID: \"bbcd7c45-75d7-43eb-a887-4eace477ec09\") "
Sep 11 18:54:46.095848 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.095797    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhr9m\" (UniqueName: \"kubernetes.io/projected/bbcd7c45-75d7-43eb-a887-4eace477ec09-kube-api-access-nhr9m\") pod \"bbcd7c45-75d7-43eb-a887-4eace477ec09\" (UID: \"bbcd7c45-75d7-43eb-a887-4eace477ec09\") "
Sep 11 18:54:46.096165 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.096137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcd7c45-75d7-43eb-a887-4eace477ec09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbcd7c45-75d7-43eb-a887-4eace477ec09" (UID: "bbcd7c45-75d7-43eb-a887-4eace477ec09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:54:46.096247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.096161    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-metrics-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:46.096247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.096182    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea330cc-51eb-4a91-b987-1c70adf4e896-maas-api-tls\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:46.096247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.096196    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea330cc-51eb-4a91-b987-1c70adf4e896-openshift-service-ca\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:46.096247 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.096210    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vz47\" (UniqueName: \"kubernetes.io/projected/5ea330cc-51eb-4a91-b987-1c70adf4e896-kube-api-access-6vz47\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:46.097779 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.097759    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcd7c45-75d7-43eb-a887-4eace477ec09-kube-api-access-nhr9m" (OuterVolumeSpecName: "kube-api-access-nhr9m") pod "bbcd7c45-75d7-43eb-a887-4eace477ec09" (UID: "bbcd7c45-75d7-43eb-a887-4eace477ec09"). InnerVolumeSpecName "kube-api-access-nhr9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:54:46.197348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.197302    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbcd7c45-75d7-43eb-a887-4eace477ec09-plugins-config-volume\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:46.197348 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.197338    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhr9m\" (UniqueName: \"kubernetes.io/projected/bbcd7c45-75d7-43eb-a887-4eace477ec09-kube-api-access-nhr9m\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:46.746220 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.746189    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"
Sep 11 18:54:46.746220 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.746208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp" event={"ID":"5ea330cc-51eb-4a91-b987-1c70adf4e896","Type":"ContainerDied","Data":"2c830ca6cd61231de7b804234fc563791b769843aefe488cd063b255a28e7d2a"}
Sep 11 18:54:46.746785 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.746277    2578 scope.go:117] "RemoveContainer" containerID="4debfd7714eee2b07beb90681112db67cdaf6b7eced2290ee1d66a76a54f3537"
Sep 11 18:54:46.747733 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.747707    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbcd7c45-75d7-43eb-a887-4eace477ec09" containerID="f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8" exitCode=0
Sep 11 18:54:46.747809 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.747755    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"
Sep 11 18:54:46.747865 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.747797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d" event={"ID":"bbcd7c45-75d7-43eb-a887-4eace477ec09","Type":"ContainerDied","Data":"f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8"}
Sep 11 18:54:46.747865 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.747851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d" event={"ID":"bbcd7c45-75d7-43eb-a887-4eace477ec09","Type":"ContainerDied","Data":"86803151e9b28a784e14996f450ec1e9a34ad7d1ed79335ef8787711e2c0b25e"}
Sep 11 18:54:46.755327 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.755170    2578 scope.go:117] "RemoveContainer" containerID="f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8"
Sep 11 18:54:46.763375 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.763354    2578 scope.go:117] "RemoveContainer" containerID="f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8"
Sep 11 18:54:46.763633 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:54:46.763617    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8\": container with ID starting with f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8 not found: ID does not exist" containerID="f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8"
Sep 11 18:54:46.763695 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.763642    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8"} err="failed to get container status \"f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8\": rpc error: code = NotFound desc = could not find container \"f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8\": container with ID starting with f5c1a29e9d4f59194e97f36fec0d34a02ae0279ae680245295707cb364aef9e8 not found: ID does not exist"
Sep 11 18:54:46.770550 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.770523    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"]
Sep 11 18:54:46.775337 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.775317    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-553671-5b7fbddd99-w2kgp"]
Sep 11 18:54:46.787410 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.787382    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"]
Sep 11 18:54:46.790994 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:46.790969    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-553671-6989bb446b-bn42d"]
Sep 11 18:54:47.937916 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:47.937881    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ea330cc-51eb-4a91-b987-1c70adf4e896" path="/var/lib/kubelet/pods/5ea330cc-51eb-4a91-b987-1c70adf4e896/volumes"
Sep 11 18:54:47.938420 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:47.938401    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbcd7c45-75d7-43eb-a887-4eace477ec09" path="/var/lib/kubelet/pods/bbcd7c45-75d7-43eb-a887-4eace477ec09/volumes"
Sep 11 18:54:55.021475 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.021441    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-9lrq9"]
Sep 11 18:54:55.021896 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.021665    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84c45b5c4-9lrq9" podUID="f0321f86-75ec-4948-b3e1-81dafebb5c20" containerName="manager" containerID="cri-o://f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2" gracePeriod=30
Sep 11 18:54:55.264709 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.264686    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:54:55.380125 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.380047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v86tb\" (UniqueName: \"kubernetes.io/projected/f0321f86-75ec-4948-b3e1-81dafebb5c20-kube-api-access-v86tb\") pod \"f0321f86-75ec-4948-b3e1-81dafebb5c20\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") "
Sep 11 18:54:55.380269 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.380123    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-webhook-cert\") pod \"f0321f86-75ec-4948-b3e1-81dafebb5c20\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") "
Sep 11 18:54:55.380269 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.380215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-metrics-certs\") pod \"f0321f86-75ec-4948-b3e1-81dafebb5c20\" (UID: \"f0321f86-75ec-4948-b3e1-81dafebb5c20\") "
Sep 11 18:54:55.382080 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.382043    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f0321f86-75ec-4948-b3e1-81dafebb5c20" (UID: "f0321f86-75ec-4948-b3e1-81dafebb5c20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:54:55.382201 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.382106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0321f86-75ec-4948-b3e1-81dafebb5c20-kube-api-access-v86tb" (OuterVolumeSpecName: "kube-api-access-v86tb") pod "f0321f86-75ec-4948-b3e1-81dafebb5c20" (UID: "f0321f86-75ec-4948-b3e1-81dafebb5c20"). InnerVolumeSpecName "kube-api-access-v86tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:54:55.382201 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.382124    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f0321f86-75ec-4948-b3e1-81dafebb5c20" (UID: "f0321f86-75ec-4948-b3e1-81dafebb5c20"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:54:55.481884 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.481850    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-metrics-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:55.481884 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.481878    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v86tb\" (UniqueName: \"kubernetes.io/projected/f0321f86-75ec-4948-b3e1-81dafebb5c20-kube-api-access-v86tb\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:55.481884 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.481888    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0321f86-75ec-4948-b3e1-81dafebb5c20-webhook-cert\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:54:55.789353 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.789322    2578 generic.go:356] "Generic (PLEG): container finished" podID="f0321f86-75ec-4948-b3e1-81dafebb5c20" containerID="f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2" exitCode=0
Sep 11 18:54:55.789530 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.789364    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c45b5c4-9lrq9"
Sep 11 18:54:55.789530 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.789410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c45b5c4-9lrq9" event={"ID":"f0321f86-75ec-4948-b3e1-81dafebb5c20","Type":"ContainerDied","Data":"f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2"}
Sep 11 18:54:55.789530 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.789461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c45b5c4-9lrq9" event={"ID":"f0321f86-75ec-4948-b3e1-81dafebb5c20","Type":"ContainerDied","Data":"987ffb7473c265c9877e6ee50515609acf65f841e6b2b137ab761e4a3293da58"}
Sep 11 18:54:55.789530 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.789484    2578 scope.go:117] "RemoveContainer" containerID="f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2"
Sep 11 18:54:55.799113 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.799095    2578 scope.go:117] "RemoveContainer" containerID="f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2"
Sep 11 18:54:55.799371 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:54:55.799355    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2\": container with ID starting with f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2 not found: ID does not exist" containerID="f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2"
Sep 11 18:54:55.799421 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.799378    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2"} err="failed to get container status \"f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2\": rpc error: code = NotFound desc = could not find container \"f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2\": container with ID starting with f9eccf39e687a6bfec2e4ace8c186ed47d1abe7d0e7e58773316a68f16b2f1b2 not found: ID does not exist"
Sep 11 18:54:55.810484 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.810462    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-9lrq9"]
Sep 11 18:54:55.814092 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.814071    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-9lrq9"]
Sep 11 18:54:55.936527 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:54:55.936488    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0321f86-75ec-4948-b3e1-81dafebb5c20" path="/var/lib/kubelet/pods/f0321f86-75ec-4948-b3e1-81dafebb5c20/volumes"
Sep 11 18:55:56.661524 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661481    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84c45b5c4-p8pg9"]
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661872    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb14558-59a5-4474-8f52-7781ca63c8f0" containerName="payload-pre-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661886    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb14558-59a5-4474-8f52-7781ca63c8f0" containerName="payload-pre-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661899    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea330cc-51eb-4a91-b987-1c70adf4e896" containerName="maas-api"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661905    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea330cc-51eb-4a91-b987-1c70adf4e896" containerName="maas-api"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661911    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2f6c27-b605-4e7f-9976-f9d50786a112" containerName="payload-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661918    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2f6c27-b605-4e7f-9976-f9d50786a112" containerName="payload-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661946    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbcd7c45-75d7-43eb-a887-4eace477ec09" containerName="payload-pre-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661951    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbcd7c45-75d7-43eb-a887-4eace477ec09" containerName="payload-pre-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661957    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a544c074-ed04-40a5-b41b-93f8cfbbfb66" containerName="payload-pre-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661963    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a544c074-ed04-40a5-b41b-93f8cfbbfb66" containerName="payload-pre-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661971    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="636e0fa1-283e-4e81-96b8-070852063a4b" containerName="payload-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661976    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="636e0fa1-283e-4e81-96b8-070852063a4b" containerName="payload-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661983    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" containerName="main"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.661988    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" containerName="main"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662001    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d26fd2-de92-4913-b317-090fdde7e740" containerName="payload-pre-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662006    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d26fd2-de92-4913-b317-090fdde7e740" containerName="payload-pre-processing"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662014    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0321f86-75ec-4948-b3e1-81dafebb5c20" containerName="manager"
Sep 11 18:55:56.662020 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662019    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0321f86-75ec-4948-b3e1-81dafebb5c20" containerName="manager"
Sep 11 18:55:56.662636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662071    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0321f86-75ec-4948-b3e1-81dafebb5c20" containerName="manager"
Sep 11 18:55:56.662636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662078    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea330cc-51eb-4a91-b987-1c70adf4e896" containerName="maas-api"
Sep 11 18:55:56.662636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662083    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbcd7c45-75d7-43eb-a887-4eace477ec09" containerName="payload-pre-processing"
Sep 11 18:55:56.662636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662089    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fb14558-59a5-4474-8f52-7781ca63c8f0" containerName="payload-pre-processing"
Sep 11 18:55:56.662636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662093    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1db525e-20cb-438f-87d0-7d8b8b3cd6ea" containerName="main"
Sep 11 18:55:56.662636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662100    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2f6c27-b605-4e7f-9976-f9d50786a112" containerName="payload-processing"
Sep 11 18:55:56.662636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662106    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a544c074-ed04-40a5-b41b-93f8cfbbfb66" containerName="payload-pre-processing"
Sep 11 18:55:56.662636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662112    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="636e0fa1-283e-4e81-96b8-070852063a4b" containerName="payload-processing"
Sep 11 18:55:56.662636 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.662117    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d26fd2-de92-4913-b317-090fdde7e740" containerName="payload-pre-processing"
Sep 11 18:55:56.665081 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.665062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:56.667597 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.667578    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7x2vx\""
Sep 11 18:55:56.668839 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.668822    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 18:55:56.668961 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.668940    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 18:55:56.669010 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.668977    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:55:56.674537 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.674513    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-p8pg9"]
Sep 11 18:55:56.809828 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.809782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-webhook-cert\") pod \"maas-controller-84c45b5c4-p8pg9\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") " pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:56.810011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.809866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtxm\" (UniqueName: \"kubernetes.io/projected/065520ae-688a-47ee-a865-3b2b19351bb6-kube-api-access-fgtxm\") pod \"maas-controller-84c45b5c4-p8pg9\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") " pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:56.810011 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.809899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-metrics-certs\") pod \"maas-controller-84c45b5c4-p8pg9\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") " pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:56.910549 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.910505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-webhook-cert\") pod \"maas-controller-84c45b5c4-p8pg9\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") " pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:56.910731 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.910568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtxm\" (UniqueName: \"kubernetes.io/projected/065520ae-688a-47ee-a865-3b2b19351bb6-kube-api-access-fgtxm\") pod \"maas-controller-84c45b5c4-p8pg9\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") " pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:56.910731 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.910592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-metrics-certs\") pod \"maas-controller-84c45b5c4-p8pg9\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") " pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:56.913083 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.913012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-metrics-certs\") pod \"maas-controller-84c45b5c4-p8pg9\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") " pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:56.913083 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.913034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-webhook-cert\") pod \"maas-controller-84c45b5c4-p8pg9\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") " pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:56.919237 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.919217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtxm\" (UniqueName: \"kubernetes.io/projected/065520ae-688a-47ee-a865-3b2b19351bb6-kube-api-access-fgtxm\") pod \"maas-controller-84c45b5c4-p8pg9\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") " pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:56.975746 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:56.975708    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:57.247519 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:57.247363    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-p8pg9"]
Sep 11 18:55:57.249753 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:55:57.249721    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065520ae_688a_47ee_a865_3b2b19351bb6.slice/crio-662b0ab7b2d86e44468fb04fb02bf84d1d3ee8a8393e4a06e0afa1a597474d9e WatchSource:0}: Error finding container 662b0ab7b2d86e44468fb04fb02bf84d1d3ee8a8393e4a06e0afa1a597474d9e: Status 404 returned error can't find the container with id 662b0ab7b2d86e44468fb04fb02bf84d1d3ee8a8393e4a06e0afa1a597474d9e
Sep 11 18:55:57.251142 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:57.251124    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:55:58.066726 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:58.066691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c45b5c4-p8pg9" event={"ID":"065520ae-688a-47ee-a865-3b2b19351bb6","Type":"ContainerStarted","Data":"4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f"}
Sep 11 18:55:58.067092 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:58.066748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c45b5c4-p8pg9" event={"ID":"065520ae-688a-47ee-a865-3b2b19351bb6","Type":"ContainerStarted","Data":"662b0ab7b2d86e44468fb04fb02bf84d1d3ee8a8393e4a06e0afa1a597474d9e"}
Sep 11 18:55:58.067092 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:58.066780    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:55:58.086136 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:55:58.086085    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84c45b5c4-p8pg9" podStartSLOduration=1.688440588 podStartE2EDuration="2.086070877s" totalImagesPullingTime="397.630289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:55:58.084219476 +0000 UTC m=+2422.663484050" watchObservedRunningTime="2026-09-11 18:55:58.086070877 +0000 UTC m=+2422.665335436"
Sep 11 18:56:09.073632 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:09.073594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:56:18.410834 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.410796    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-p8pg9"]
Sep 11 18:56:18.411296 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.411006    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84c45b5c4-p8pg9" podUID="065520ae-688a-47ee-a865-3b2b19351bb6" containerName="manager" containerID="cri-o://4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f" gracePeriod=30
Sep 11 18:56:18.652474 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.652450    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:56:18.712714 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.712625    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-webhook-cert\") pod \"065520ae-688a-47ee-a865-3b2b19351bb6\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") "
Sep 11 18:56:18.712714 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.712711    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgtxm\" (UniqueName: \"kubernetes.io/projected/065520ae-688a-47ee-a865-3b2b19351bb6-kube-api-access-fgtxm\") pod \"065520ae-688a-47ee-a865-3b2b19351bb6\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") "
Sep 11 18:56:18.712888 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.712731    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-metrics-certs\") pod \"065520ae-688a-47ee-a865-3b2b19351bb6\" (UID: \"065520ae-688a-47ee-a865-3b2b19351bb6\") "
Sep 11 18:56:18.714608 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.714572    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "065520ae-688a-47ee-a865-3b2b19351bb6" (UID: "065520ae-688a-47ee-a865-3b2b19351bb6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:56:18.714804 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.714759    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "065520ae-688a-47ee-a865-3b2b19351bb6" (UID: "065520ae-688a-47ee-a865-3b2b19351bb6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:56:18.714856 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.714835    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065520ae-688a-47ee-a865-3b2b19351bb6-kube-api-access-fgtxm" (OuterVolumeSpecName: "kube-api-access-fgtxm") pod "065520ae-688a-47ee-a865-3b2b19351bb6" (UID: "065520ae-688a-47ee-a865-3b2b19351bb6"). InnerVolumeSpecName "kube-api-access-fgtxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:56:18.814155 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.814121    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-webhook-cert\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:56:18.814155 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.814154    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtxm\" (UniqueName: \"kubernetes.io/projected/065520ae-688a-47ee-a865-3b2b19351bb6-kube-api-access-fgtxm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:56:18.814155 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:18.814163    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/065520ae-688a-47ee-a865-3b2b19351bb6-metrics-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 18:56:19.159349 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:19.159306    2578 generic.go:356] "Generic (PLEG): container finished" podID="065520ae-688a-47ee-a865-3b2b19351bb6" containerID="4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f" exitCode=0
Sep 11 18:56:19.159526 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:19.159355    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c45b5c4-p8pg9"
Sep 11 18:56:19.159526 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:19.159387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c45b5c4-p8pg9" event={"ID":"065520ae-688a-47ee-a865-3b2b19351bb6","Type":"ContainerDied","Data":"4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f"}
Sep 11 18:56:19.159526 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:19.159440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c45b5c4-p8pg9" event={"ID":"065520ae-688a-47ee-a865-3b2b19351bb6","Type":"ContainerDied","Data":"662b0ab7b2d86e44468fb04fb02bf84d1d3ee8a8393e4a06e0afa1a597474d9e"}
Sep 11 18:56:19.159526 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:19.159460    2578 scope.go:117] "RemoveContainer" containerID="4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f"
Sep 11 18:56:19.168390 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:19.168373    2578 scope.go:117] "RemoveContainer" containerID="4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f"
Sep 11 18:56:19.168641 ip-10-0-139-186 kubenswrapper[2578]: E0911 18:56:19.168622    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f\": container with ID starting with 4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f not found: ID does not exist" containerID="4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f"
Sep 11 18:56:19.168716 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:19.168663    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f"} err="failed to get container status \"4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f\": rpc error: code = NotFound desc = could not find container \"4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f\": container with ID starting with 4443cb85436363b91f2a48594640578e09e263e4b704a479f5c154b57fef8c9f not found: ID does not exist"
Sep 11 18:56:19.180293 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:19.180270    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-p8pg9"]
Sep 11 18:56:19.184006 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:19.183986    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-p8pg9"]
Sep 11 18:56:19.937362 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:19.937326    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065520ae-688a-47ee-a865-3b2b19351bb6" path="/var/lib/kubelet/pods/065520ae-688a-47ee-a865-3b2b19351bb6/volumes"
Sep 11 18:56:20.007790 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.007756    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84c45b5c4-54hmv"]
Sep 11 18:56:20.008092 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.008080    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065520ae-688a-47ee-a865-3b2b19351bb6" containerName="manager"
Sep 11 18:56:20.008148 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.008094    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="065520ae-688a-47ee-a865-3b2b19351bb6" containerName="manager"
Sep 11 18:56:20.008184 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.008157    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="065520ae-688a-47ee-a865-3b2b19351bb6" containerName="manager"
Sep 11 18:56:20.012322 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.012305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.015161 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.015143    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:56:20.016239 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.016217    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 18:56:20.016335 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.016297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7x2vx\""
Sep 11 18:56:20.016335 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.016302    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 18:56:20.025900 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.025866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8f64\" (UniqueName: \"kubernetes.io/projected/d42ec6cc-31fc-4d85-891a-fb5e2fb486dc-kube-api-access-r8f64\") pod \"maas-controller-84c45b5c4-54hmv\" (UID: \"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc\") " pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.026046 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.025995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42ec6cc-31fc-4d85-891a-fb5e2fb486dc-metrics-certs\") pod \"maas-controller-84c45b5c4-54hmv\" (UID: \"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc\") " pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.026410 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.026382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d42ec6cc-31fc-4d85-891a-fb5e2fb486dc-webhook-cert\") pod \"maas-controller-84c45b5c4-54hmv\" (UID: \"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc\") " pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.028143 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.028119    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-54hmv"]
Sep 11 18:56:20.126822 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.126795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8f64\" (UniqueName: \"kubernetes.io/projected/d42ec6cc-31fc-4d85-891a-fb5e2fb486dc-kube-api-access-r8f64\") pod \"maas-controller-84c45b5c4-54hmv\" (UID: \"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc\") " pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.126958 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.126839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42ec6cc-31fc-4d85-891a-fb5e2fb486dc-metrics-certs\") pod \"maas-controller-84c45b5c4-54hmv\" (UID: \"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc\") " pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.126958 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.126887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d42ec6cc-31fc-4d85-891a-fb5e2fb486dc-webhook-cert\") pod \"maas-controller-84c45b5c4-54hmv\" (UID: \"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc\") " pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.129328 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.129299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d42ec6cc-31fc-4d85-891a-fb5e2fb486dc-webhook-cert\") pod \"maas-controller-84c45b5c4-54hmv\" (UID: \"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc\") " pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.129453 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.129368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42ec6cc-31fc-4d85-891a-fb5e2fb486dc-metrics-certs\") pod \"maas-controller-84c45b5c4-54hmv\" (UID: \"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc\") " pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.135951 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.135918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8f64\" (UniqueName: \"kubernetes.io/projected/d42ec6cc-31fc-4d85-891a-fb5e2fb486dc-kube-api-access-r8f64\") pod \"maas-controller-84c45b5c4-54hmv\" (UID: \"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc\") " pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.329427 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.329391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:20.600311 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:20.600281    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c45b5c4-54hmv"]
Sep 11 18:56:20.602831 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:56:20.602792    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42ec6cc_31fc_4d85_891a_fb5e2fb486dc.slice/crio-4c6feac0de233472cde62c0f62ad4d326934864a88d75619d03e0ca81f3f1373 WatchSource:0}: Error finding container 4c6feac0de233472cde62c0f62ad4d326934864a88d75619d03e0ca81f3f1373: Status 404 returned error can't find the container with id 4c6feac0de233472cde62c0f62ad4d326934864a88d75619d03e0ca81f3f1373
Sep 11 18:56:21.170227 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:21.170184    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c45b5c4-54hmv" event={"ID":"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc","Type":"ContainerStarted","Data":"68a397472a25f93ceddf23c3caf30119b248d1da9c49400d5fcc60b6e83f68e2"}
Sep 11 18:56:21.170717 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:21.170246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c45b5c4-54hmv" event={"ID":"d42ec6cc-31fc-4d85-891a-fb5e2fb486dc","Type":"ContainerStarted","Data":"4c6feac0de233472cde62c0f62ad4d326934864a88d75619d03e0ca81f3f1373"}
Sep 11 18:56:21.170717 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:21.170385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:21.189202 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:21.189096    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84c45b5c4-54hmv" podStartSLOduration=1.856147568 podStartE2EDuration="2.189082143s" totalImagesPullingTime="332.934575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:56:21.187240505 +0000 UTC m=+2445.766505064" watchObservedRunningTime="2026-09-11 18:56:21.189082143 +0000 UTC m=+2445.768346757"
Sep 11 18:56:32.177516 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:32.177474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84c45b5c4-54hmv"
Sep 11 18:56:36.331204 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:56:36.331169    2578 scope.go:117] "RemoveContainer" containerID="3f7144be4abb40450d16f3ebb48ff79f79928f7231faa06481d606d970637f87"
Sep 11 18:59:37.947436 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:37.947404    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"]
Sep 11 18:59:37.950902 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:37.950872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:37.953622 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:37.953590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c6d92ea4-kserve-self-signed-certs\""
Sep 11 18:59:37.959536 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:37.959505    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"]
Sep 11 18:59:38.110906 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.110867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-model-cache\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.111135 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.110927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-tmp-dir\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.111135 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.110971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-home\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.111135 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.111048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-dshm\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.111298 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.111145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7pg\" (UniqueName: \"kubernetes.io/projected/86de6f40-0999-431c-a91e-261607e8be5b-kube-api-access-px7pg\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.111298 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.111173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86de6f40-0999-431c-a91e-261607e8be5b-tls-certs\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.212120 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.212026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86de6f40-0999-431c-a91e-261607e8be5b-tls-certs\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.212120 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.212082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-model-cache\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.212120 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.212114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-tmp-dir\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.212416 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.212141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-home\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.212416 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.212186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-dshm\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.212416 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.212279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7pg\" (UniqueName: \"kubernetes.io/projected/86de6f40-0999-431c-a91e-261607e8be5b-kube-api-access-px7pg\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.212664 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.212621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-model-cache\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.212759 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.212668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-home\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.212759 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.212711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-tmp-dir\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.214574 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.214551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-dshm\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.214734 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.214718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86de6f40-0999-431c-a91e-261607e8be5b-tls-certs\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.220185 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.220163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7pg\" (UniqueName: \"kubernetes.io/projected/86de6f40-0999-431c-a91e-261607e8be5b-kube-api-access-px7pg\") pod \"e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") " pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.264268 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.264228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:38.541467 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:38.541400    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"]
Sep 11 18:59:38.543971 ip-10-0-139-186 kubenswrapper[2578]: W0911 18:59:38.543941    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86de6f40_0999_431c_a91e_261607e8be5b.slice/crio-436315e11ea0a2e43fd65c5e31e550222b5309921b91504a7a9527c8b3dcc2f6 WatchSource:0}: Error finding container 436315e11ea0a2e43fd65c5e31e550222b5309921b91504a7a9527c8b3dcc2f6: Status 404 returned error can't find the container with id 436315e11ea0a2e43fd65c5e31e550222b5309921b91504a7a9527c8b3dcc2f6
Sep 11 18:59:39.076686 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:39.076641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg" event={"ID":"86de6f40-0999-431c-a91e-261607e8be5b","Type":"ContainerStarted","Data":"c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064"}
Sep 11 18:59:39.077053 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:39.076695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg" event={"ID":"86de6f40-0999-431c-a91e-261607e8be5b","Type":"ContainerStarted","Data":"436315e11ea0a2e43fd65c5e31e550222b5309921b91504a7a9527c8b3dcc2f6"}
Sep 11 18:59:39.095434 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:39.095377    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg" podStartSLOduration=2.095360353 podStartE2EDuration="2.095360353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:59:39.093606145 +0000 UTC m=+2643.672870731" watchObservedRunningTime="2026-09-11 18:59:39.095360353 +0000 UTC m=+2643.674624914"
Sep 11 18:59:48.265016 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:48.264972    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:48.265450 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:48.265060    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:48.277444 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:48.277418    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:49.132910 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:49.132876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 18:59:56.569023 ip-10-0-139-186 kubenswrapper[2578]: I0911 18:59:56.568990    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"]
Sep 11 19:00:11.589982 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.589908    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg" podUID="86de6f40-0999-431c-a91e-261607e8be5b" containerName="main" containerID="cri-o://c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064" gracePeriod=45
Sep 11 19:00:11.830471 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.830445    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 19:00:11.914262 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914186    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-model-cache\") pod \"86de6f40-0999-431c-a91e-261607e8be5b\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") "
Sep 11 19:00:11.914415 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px7pg\" (UniqueName: \"kubernetes.io/projected/86de6f40-0999-431c-a91e-261607e8be5b-kube-api-access-px7pg\") pod \"86de6f40-0999-431c-a91e-261607e8be5b\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") "
Sep 11 19:00:11.914415 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914316    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-dshm\") pod \"86de6f40-0999-431c-a91e-261607e8be5b\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") "
Sep 11 19:00:11.914415 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-tmp-dir\") pod \"86de6f40-0999-431c-a91e-261607e8be5b\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") "
Sep 11 19:00:11.914415 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-home\") pod \"86de6f40-0999-431c-a91e-261607e8be5b\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") "
Sep 11 19:00:11.914415 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86de6f40-0999-431c-a91e-261607e8be5b-tls-certs\") pod \"86de6f40-0999-431c-a91e-261607e8be5b\" (UID: \"86de6f40-0999-431c-a91e-261607e8be5b\") "
Sep 11 19:00:11.914720 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "86de6f40-0999-431c-a91e-261607e8be5b" (UID: "86de6f40-0999-431c-a91e-261607e8be5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:00:11.914720 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914609    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86de6f40-0999-431c-a91e-261607e8be5b" (UID: "86de6f40-0999-431c-a91e-261607e8be5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:00:11.914720 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-home" (OuterVolumeSpecName: "home") pod "86de6f40-0999-431c-a91e-261607e8be5b" (UID: "86de6f40-0999-431c-a91e-261607e8be5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:00:11.914720 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914713    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-model-cache\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 19:00:11.914866 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.914726    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-tmp-dir\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 19:00:11.916566 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.916532    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86de6f40-0999-431c-a91e-261607e8be5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86de6f40-0999-431c-a91e-261607e8be5b" (UID: "86de6f40-0999-431c-a91e-261607e8be5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:00:11.916692 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.916564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-dshm" (OuterVolumeSpecName: "dshm") pod "86de6f40-0999-431c-a91e-261607e8be5b" (UID: "86de6f40-0999-431c-a91e-261607e8be5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:00:11.916692 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:11.916577    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86de6f40-0999-431c-a91e-261607e8be5b-kube-api-access-px7pg" (OuterVolumeSpecName: "kube-api-access-px7pg") pod "86de6f40-0999-431c-a91e-261607e8be5b" (UID: "86de6f40-0999-431c-a91e-261607e8be5b"). InnerVolumeSpecName "kube-api-access-px7pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:00:12.015612 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.015590    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-dshm\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 19:00:12.015612 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.015609    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86de6f40-0999-431c-a91e-261607e8be5b-home\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 19:00:12.015762 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.015618    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86de6f40-0999-431c-a91e-261607e8be5b-tls-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 19:00:12.015762 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.015627    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px7pg\" (UniqueName: \"kubernetes.io/projected/86de6f40-0999-431c-a91e-261607e8be5b-kube-api-access-px7pg\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 19:00:12.223696 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.223610    2578 generic.go:356] "Generic (PLEG): container finished" podID="86de6f40-0999-431c-a91e-261607e8be5b" containerID="c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064" exitCode=0
Sep 11 19:00:12.223696 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.223667    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"
Sep 11 19:00:12.223885 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.223700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg" event={"ID":"86de6f40-0999-431c-a91e-261607e8be5b","Type":"ContainerDied","Data":"c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064"}
Sep 11 19:00:12.223885 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.223751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg" event={"ID":"86de6f40-0999-431c-a91e-261607e8be5b","Type":"ContainerDied","Data":"436315e11ea0a2e43fd65c5e31e550222b5309921b91504a7a9527c8b3dcc2f6"}
Sep 11 19:00:12.223885 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.223776    2578 scope.go:117] "RemoveContainer" containerID="c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064"
Sep 11 19:00:12.231782 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.231767    2578 scope.go:117] "RemoveContainer" containerID="c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064"
Sep 11 19:00:12.232070 ip-10-0-139-186 kubenswrapper[2578]: E0911 19:00:12.232050    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064\": container with ID starting with c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064 not found: ID does not exist" containerID="c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064"
Sep 11 19:00:12.232145 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.232087    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064"} err="failed to get container status \"c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064\": rpc error: code = NotFound desc = could not find container \"c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064\": container with ID starting with c0a4e25601f4399f71552c3acefbc95d9a827865da1a9f19eb42af5641b08064 not found: ID does not exist"
Sep 11 19:00:12.243038 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.243015    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"]
Sep 11 19:00:12.247483 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:12.247463    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c6d92ea4-kserve-ffcfb764c-f2cqg"]
Sep 11 19:00:13.937438 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:13.937364    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86de6f40-0999-431c-a91e-261607e8be5b" path="/var/lib/kubelet/pods/86de6f40-0999-431c-a91e-261607e8be5b/volumes"
Sep 11 19:00:27.827239 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:27.827197    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"]
Sep 11 19:00:27.827786 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:27.827764    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86de6f40-0999-431c-a91e-261607e8be5b" containerName="main"
Sep 11 19:00:27.827862 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:27.827789    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="86de6f40-0999-431c-a91e-261607e8be5b" containerName="main"
Sep 11 19:00:27.827922 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:27.827899    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="86de6f40-0999-431c-a91e-261607e8be5b" containerName="main"
Sep 11 19:00:27.830979 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:27.830956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:27.841555 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:27.841524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"]
Sep 11 19:00:27.951644 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:27.951544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67r6g\" (UniqueName: \"kubernetes.io/projected/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-kube-api-access-67r6g\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:27.951644 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:27.951590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-openshift-service-ca\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:27.951644 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:27.951621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-maas-api-tls\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:27.951894 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:27.951728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-metrics-certs\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:28.053052 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:28.053013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67r6g\" (UniqueName: \"kubernetes.io/projected/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-kube-api-access-67r6g\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:28.053052 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:28.053062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-openshift-service-ca\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:28.053321 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:28.053109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-maas-api-tls\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:28.053321 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:28.053180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-metrics-certs\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:28.053992 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:28.053962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-openshift-service-ca\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:28.055739 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:28.055716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-maas-api-tls\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:28.055853 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:28.055831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-metrics-certs\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:28.062231 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:28.062203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67r6g\" (UniqueName: \"kubernetes.io/projected/7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11-kube-api-access-67r6g\") pod \"maas-api-765c4fc4c8-sr6vn\" (UID: \"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11\") " pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:28.142981 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:28.142869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:28.420264 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:28.420232    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"]
Sep 11 19:00:28.422530 ip-10-0-139-186 kubenswrapper[2578]: W0911 19:00:28.422493    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba6a12d_4dbd_4bd3_a6ad_6cc569f76e11.slice/crio-e85cfe6236d503f45734fdf4a3536f2345bf587d52ba6ef79abbd6cf8131aef4 WatchSource:0}: Error finding container e85cfe6236d503f45734fdf4a3536f2345bf587d52ba6ef79abbd6cf8131aef4: Status 404 returned error can't find the container with id e85cfe6236d503f45734fdf4a3536f2345bf587d52ba6ef79abbd6cf8131aef4
Sep 11 19:00:29.304273 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:29.304235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn" event={"ID":"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11","Type":"ContainerStarted","Data":"48a77664febca35989e39c4121b7df25d210be1758d7f84c23c2436e28669c41"}
Sep 11 19:00:29.304725 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:29.304296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn" event={"ID":"7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11","Type":"ContainerStarted","Data":"e85cfe6236d503f45734fdf4a3536f2345bf587d52ba6ef79abbd6cf8131aef4"}
Sep 11 19:00:29.304725 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:29.304375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:29.323148 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:29.323093    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn" podStartSLOduration=1.953688071 podStartE2EDuration="2.323076064s" totalImagesPullingTime="369.387993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:29.321706608 +0000 UTC m=+2693.900971182" watchObservedRunningTime="2026-09-11 19:00:29.323076064 +0000 UTC m=+2693.902340663"
Sep 11 19:00:35.313821 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:35.313783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-765c4fc4c8-sr6vn"
Sep 11 19:00:35.361870 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:35.361836    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-b97cd"]
Sep 11 19:00:35.362089 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:35.362051    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd" podUID="161921c5-e475-41b2-a9e9-dc711195e82a" containerName="maas-api" containerID="cri-o://98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c" gracePeriod=30
Sep 11 19:00:36.613079 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.613051    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 19:00:36.736076 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.735976    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-metrics-certs\") pod \"161921c5-e475-41b2-a9e9-dc711195e82a\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") "
Sep 11 19:00:36.736076 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.736031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-maas-api-tls\") pod \"161921c5-e475-41b2-a9e9-dc711195e82a\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") "
Sep 11 19:00:36.736076 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.736071    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/161921c5-e475-41b2-a9e9-dc711195e82a-openshift-service-ca\") pod \"161921c5-e475-41b2-a9e9-dc711195e82a\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") "
Sep 11 19:00:36.736323 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.736144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm6sf\" (UniqueName: \"kubernetes.io/projected/161921c5-e475-41b2-a9e9-dc711195e82a-kube-api-access-cm6sf\") pod \"161921c5-e475-41b2-a9e9-dc711195e82a\" (UID: \"161921c5-e475-41b2-a9e9-dc711195e82a\") "
Sep 11 19:00:36.736500 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.736477    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161921c5-e475-41b2-a9e9-dc711195e82a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "161921c5-e475-41b2-a9e9-dc711195e82a" (UID: "161921c5-e475-41b2-a9e9-dc711195e82a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:00:36.738147 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.738121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "161921c5-e475-41b2-a9e9-dc711195e82a" (UID: "161921c5-e475-41b2-a9e9-dc711195e82a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:00:36.738276 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.738258    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "161921c5-e475-41b2-a9e9-dc711195e82a" (UID: "161921c5-e475-41b2-a9e9-dc711195e82a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:00:36.738325 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.738270    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161921c5-e475-41b2-a9e9-dc711195e82a-kube-api-access-cm6sf" (OuterVolumeSpecName: "kube-api-access-cm6sf") pod "161921c5-e475-41b2-a9e9-dc711195e82a" (UID: "161921c5-e475-41b2-a9e9-dc711195e82a"). InnerVolumeSpecName "kube-api-access-cm6sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:00:36.836867 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.836824    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-metrics-certs\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 19:00:36.836867 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.836859    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/161921c5-e475-41b2-a9e9-dc711195e82a-maas-api-tls\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 19:00:36.836867 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.836869    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/161921c5-e475-41b2-a9e9-dc711195e82a-openshift-service-ca\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 19:00:36.837114 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:36.836888    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm6sf\" (UniqueName: \"kubernetes.io/projected/161921c5-e475-41b2-a9e9-dc711195e82a-kube-api-access-cm6sf\") on node \"ip-10-0-139-186.ec2.internal\" DevicePath \"\""
Sep 11 19:00:37.343022 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:37.342988    2578 generic.go:356] "Generic (PLEG): container finished" podID="161921c5-e475-41b2-a9e9-dc711195e82a" containerID="98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c" exitCode=0
Sep 11 19:00:37.343178 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:37.343052    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd"
Sep 11 19:00:37.343178 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:37.343068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd" event={"ID":"161921c5-e475-41b2-a9e9-dc711195e82a","Type":"ContainerDied","Data":"98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c"}
Sep 11 19:00:37.343178 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:37.343132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976f486c4-b97cd" event={"ID":"161921c5-e475-41b2-a9e9-dc711195e82a","Type":"ContainerDied","Data":"3ef52d4d7f6fca79693f9fbb49ed7c3d6fc68d1c0e664b97c46a75a06a944e99"}
Sep 11 19:00:37.343178 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:37.343158    2578 scope.go:117] "RemoveContainer" containerID="98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c"
Sep 11 19:00:37.353757 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:37.353638    2578 scope.go:117] "RemoveContainer" containerID="98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c"
Sep 11 19:00:37.354064 ip-10-0-139-186 kubenswrapper[2578]: E0911 19:00:37.354045    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c\": container with ID starting with 98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c not found: ID does not exist" containerID="98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c"
Sep 11 19:00:37.354121 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:37.354074    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c"} err="failed to get container status \"98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c\": rpc error: code = NotFound desc = could not find container \"98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c\": container with ID starting with 98f3e2c4e808025f31cb5a3a542a4f7e664d64acb71507e9e8b91201ae41579c not found: ID does not exist"
Sep 11 19:00:37.367408 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:37.367380    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-b97cd"]
Sep 11 19:00:37.373641 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:37.373612    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-976f486c4-b97cd"]
Sep 11 19:00:37.937619 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:37.937583    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="161921c5-e475-41b2-a9e9-dc711195e82a" path="/var/lib/kubelet/pods/161921c5-e475-41b2-a9e9-dc711195e82a/volumes"
Sep 11 19:00:55.807005 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:55.806962    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84c45b5c4-54hmv_d42ec6cc-31fc-4d85-891a-fb5e2fb486dc/manager/0.log"
Sep 11 19:00:56.187860 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:56.187778    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b7b6b756-bzkzq_d8a9b66c-2e36-405d-8d4a-78c24560f6fb/copy-manifests/0.log"
Sep 11 19:00:56.194550 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:56.194525    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b7b6b756-bzkzq_d8a9b66c-2e36-405d-8d4a-78c24560f6fb/manager/0.log"
Sep 11 19:00:56.805313 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:56.805282    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-765c4fc4c8-sr6vn_7ba6a12d-4dbd-4bd3-a6ad-6cc569f76e11/maas-api/0.log"
Sep 11 19:00:57.354132 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:57.354104    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-vlmmq_9f4b16a2-88d3-4687-b55a-9fe04887351c/postgres/0.log"
Sep 11 19:00:59.502824 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:59.502791    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58fb58cc-n4mp7_ff434832-a415-4153-bd86-1cbace75a7a5/kube-auth-proxy/0.log"
Sep 11 19:00:59.764985 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:00:59.764902    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-mh2b4_2a4ae1ca-0b57-492c-b6f1-dba25973b579/payload-pre-processing/0.log"
Sep 11 19:01:00.461824 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:00.461794    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-sjkxb_819416f3-065f-430e-b277-8f718459feac/main/0.log"
Sep 11 19:01:00.576518 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:00.576485    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-szmtm_54b6cb06-0c4b-4825-9dac-9b32336a882c/main/0.log"
Sep 11 19:01:00.804682 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:00.804639    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbbwzj_66397def-0642-484c-9ad9-b0c635ee8385/main/0.log"
Sep 11 19:01:10.014392 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:10.014351    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-65zrr_2050106a-dd99-409d-9860-6887d827d3af/global-pull-secret-syncer/0.log"
Sep 11 19:01:10.275234 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:10.275205    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ztzmx_9129c4f7-d49b-471e-abe5-81dbfdce82eb/konnectivity-agent/0.log"
Sep 11 19:01:10.299691 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:10.299638    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-186.ec2.internal_14d74969f57bbabba180bdcaf49f1bc5/haproxy/0.log"
Sep 11 19:01:16.723785 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:16.723755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6c7c4c7659-dn7nz_e5d7b414-8d85-4313-a7cd-52b7afb54ae8/metrics-server/0.log"
Sep 11 19:01:17.004613 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.004581    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wlv42_6fe2ae64-a407-4ecb-b22b-3e4ff72ca409/node-exporter/0.log"
Sep 11 19:01:17.037096 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.037066    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wlv42_6fe2ae64-a407-4ecb-b22b-3e4ff72ca409/kube-rbac-proxy/0.log"
Sep 11 19:01:17.067796 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.067764    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wlv42_6fe2ae64-a407-4ecb-b22b-3e4ff72ca409/init-textfile/0.log"
Sep 11 19:01:17.258940 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.258900    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70baba0f-ad60-47f2-b499-0c40808bf6a6/prometheus/0.log"
Sep 11 19:01:17.310775 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.310743    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70baba0f-ad60-47f2-b499-0c40808bf6a6/config-reloader/0.log"
Sep 11 19:01:17.347390 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.347367    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70baba0f-ad60-47f2-b499-0c40808bf6a6/thanos-sidecar/0.log"
Sep 11 19:01:17.382951 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.382924    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70baba0f-ad60-47f2-b499-0c40808bf6a6/kube-rbac-proxy-web/0.log"
Sep 11 19:01:17.418725 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.417670    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70baba0f-ad60-47f2-b499-0c40808bf6a6/kube-rbac-proxy/0.log"
Sep 11 19:01:17.457703 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.457583    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70baba0f-ad60-47f2-b499-0c40808bf6a6/kube-rbac-proxy-thanos/0.log"
Sep 11 19:01:17.569506 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.569469    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70baba0f-ad60-47f2-b499-0c40808bf6a6/init-config-reloader/0.log"
Sep 11 19:01:17.732850 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.732763    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-7tcxj_c35d6880-51f7-454f-a1bd-a1fe822a6f79/prometheus-operator/0.log"
Sep 11 19:01:17.804467 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:17.804415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-7tcxj_c35d6880-51f7-454f-a1bd-a1fe822a6f79/kube-rbac-proxy/0.log"
Sep 11 19:01:18.064343 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:18.064310    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b5db496-jrr8p_d5b7611c-a626-41d6-82f8-080815c87c27/thanos-query/0.log"
Sep 11 19:01:18.105174 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:18.105146    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b5db496-jrr8p_d5b7611c-a626-41d6-82f8-080815c87c27/kube-rbac-proxy-web/0.log"
Sep 11 19:01:18.150742 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:18.150711    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b5db496-jrr8p_d5b7611c-a626-41d6-82f8-080815c87c27/kube-rbac-proxy/0.log"
Sep 11 19:01:18.177453 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:18.177420    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b5db496-jrr8p_d5b7611c-a626-41d6-82f8-080815c87c27/prom-label-proxy/0.log"
Sep 11 19:01:18.200946 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:18.200921    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b5db496-jrr8p_d5b7611c-a626-41d6-82f8-080815c87c27/kube-rbac-proxy-rules/0.log"
Sep 11 19:01:18.247818 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:18.247790    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-588b5db496-jrr8p_d5b7611c-a626-41d6-82f8-080815c87c27/kube-rbac-proxy-metrics/0.log"
Sep 11 19:01:20.130000 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.129961    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"]
Sep 11 19:01:20.130434 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.130387    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161921c5-e475-41b2-a9e9-dc711195e82a" containerName="maas-api"
Sep 11 19:01:20.130434 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.130402    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="161921c5-e475-41b2-a9e9-dc711195e82a" containerName="maas-api"
Sep 11 19:01:20.130520 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.130485    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="161921c5-e475-41b2-a9e9-dc711195e82a" containerName="maas-api"
Sep 11 19:01:20.133425 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.133408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.136180 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.136158    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5r55n\"/\"default-dockercfg-rblx9\""
Sep 11 19:01:20.136388 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.136371    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5r55n\"/\"openshift-service-ca.crt\""
Sep 11 19:01:20.136443 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.136392    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5r55n\"/\"kube-root-ca.crt\""
Sep 11 19:01:20.143752 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.143726    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"]
Sep 11 19:01:20.243897 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.243855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-podres\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.244082 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.243990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-sys\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.244082 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.244020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-proc\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.244082 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.244039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-lib-modules\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.244082 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.244060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dph9g\" (UniqueName: \"kubernetes.io/projected/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-kube-api-access-dph9g\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.344798 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.344758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-sys\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.344798 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.344799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-proc\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.344798 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.344818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-lib-modules\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.345098 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.344837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dph9g\" (UniqueName: \"kubernetes.io/projected/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-kube-api-access-dph9g\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.345098 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.344905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-podres\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.345098 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.344905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-sys\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.345098 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.344912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-proc\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.345098 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.344975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-lib-modules\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.345098 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.345014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-podres\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.366935 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.366900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dph9g\" (UniqueName: \"kubernetes.io/projected/d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e-kube-api-access-dph9g\") pod \"perf-node-gather-daemonset-tc5mr\" (UID: \"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e\") " pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.444420 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.444325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:20.719408 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:20.719380    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"]
Sep 11 19:01:20.721490 ip-10-0-139-186 kubenswrapper[2578]: W0911 19:01:20.721460    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd0b0cfdd_a87b_4dd6_92c5_57dcd2aa867e.slice/crio-d0cc982b0974da00c3f571063c6568f3efdee7b39b6a7407d793f80cb6c80a13 WatchSource:0}: Error finding container d0cc982b0974da00c3f571063c6568f3efdee7b39b6a7407d793f80cb6c80a13: Status 404 returned error can't find the container with id d0cc982b0974da00c3f571063c6568f3efdee7b39b6a7407d793f80cb6c80a13
Sep 11 19:01:21.497607 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:21.497570    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-4429d_ac502d25-97f4-4c61-9725-93b13caf9437/volume-data-source-validator/0.log"
Sep 11 19:01:21.550878 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:21.550836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr" event={"ID":"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e","Type":"ContainerStarted","Data":"145fbe6e45348e69462b3f40fe88d9f13fcf4671f1bb3c9d87933093d17ab4e5"}
Sep 11 19:01:21.551034 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:21.550903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr" event={"ID":"d0b0cfdd-a87b-4dd6-92c5-57dcd2aa867e","Type":"ContainerStarted","Data":"d0cc982b0974da00c3f571063c6568f3efdee7b39b6a7407d793f80cb6c80a13"}
Sep 11 19:01:21.551096 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:21.551054    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:21.568747 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:21.568696    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr" podStartSLOduration=1.568676585 podStartE2EDuration="1.568676585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:01:21.567457433 +0000 UTC m=+2746.146721994" watchObservedRunningTime="2026-09-11 19:01:21.568676585 +0000 UTC m=+2746.147941147"
Sep 11 19:01:22.643456 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:22.643416    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tjgtz_6543db77-6046-4e91-9420-88dba053c7f3/dns/0.log"
Sep 11 19:01:22.665359 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:22.665328    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tjgtz_6543db77-6046-4e91-9420-88dba053c7f3/kube-rbac-proxy/0.log"
Sep 11 19:01:22.689715 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:22.689686    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2x9c6_101bbe73-40f7-41b8-89a2-b09ba7ef75ca/dns-node-resolver/0.log"
Sep 11 19:01:23.432307 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:23.432275    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l4h5q_861c7885-6882-4d54-a19a-99b7c86dc58e/node-ca/0.log"
Sep 11 19:01:25.054201 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:25.054171    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58fb58cc-n4mp7_ff434832-a415-4153-bd86-1cbace75a7a5/kube-auth-proxy/0.log"
Sep 11 19:01:25.119472 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:25.119439    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-mh2b4_2a4ae1ca-0b57-492c-b6f1-dba25973b579/payload-pre-processing/0.log"
Sep 11 19:01:26.152429 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:26.152399    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kd5hz_7d944d63-7ebc-46d4-8a4d-3b7d21e29c7d/serve-healthcheck-canary/0.log"
Sep 11 19:01:26.646170 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:26.646136    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-578v7_509846c6-e854-4e12-be62-b604e7b76faa/kube-rbac-proxy/0.log"
Sep 11 19:01:26.667419 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:26.667387    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-578v7_509846c6-e854-4e12-be62-b604e7b76faa/exporter/0.log"
Sep 11 19:01:26.688123 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:26.688095    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-578v7_509846c6-e854-4e12-be62-b604e7b76faa/extractor/0.log"
Sep 11 19:01:27.566019 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:27.565991    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5r55n/perf-node-gather-daemonset-tc5mr"
Sep 11 19:01:29.357223 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:29.357182    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84c45b5c4-54hmv_d42ec6cc-31fc-4d85-891a-fb5e2fb486dc/manager/0.log"
Sep 11 19:01:29.575468 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:29.575438    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b7b6b756-bzkzq_d8a9b66c-2e36-405d-8d4a-78c24560f6fb/manager/0.log"
Sep 11 19:01:29.605483 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:29.605450    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b7b6b756-bzkzq_d8a9b66c-2e36-405d-8d4a-78c24560f6fb/copy-manifests/0.log"
Sep 11 19:01:31.202884 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:31.202850    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78b8f7c5cb-8ncpw_6a843932-4f80-486d-81ec-a561a802469b/manager/0.log"
Sep 11 19:01:31.544570 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:31.544541    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58fb58cc-n4mp7_ff434832-a415-4153-bd86-1cbace75a7a5/kube-auth-proxy/0.log"
Sep 11 19:01:31.599899 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:31.599869    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-mh2b4_2a4ae1ca-0b57-492c-b6f1-dba25973b579/payload-pre-processing/0.log"
Sep 11 19:01:33.689659 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:33.689629    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58fb58cc-n4mp7_ff434832-a415-4153-bd86-1cbace75a7a5/kube-auth-proxy/0.log"
Sep 11 19:01:33.741148 ip-10-0-139-186 kubenswrapper[2578]: I0911 19:01:33.741106    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-mh2b4_2a4ae1ca-0b57-492c-b6f1-dba25973b579/payload-pre-processing/0.log"